Support Center

Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Wrong category when import Palo Alto from a log file

Max Monterumisi Jan 03, 2017 07:33AM PST

Hi,
from Palo Alto -> URL Filtering log I had exported data on a csv file then I had imported on new Storage on Vantage Ultimate v.2.2.0.199
Before build a report, I take a look with Summaries Ad-Hoc analysis.
Inside the analysis, in the CATEGORY, together with good categories of Palo Alto, I see MANY More wrong categories.
For example category with name: "11243901" or "inQuirkMode:"False}(9999)"

I went in deep on log file an this values come from other field of the csv file (ex: seqno)
I guess this means the import process try some bad charter on the log file and fail to import the correct field on the correct place.

For example, the bad category value 11243901 appear only the following csv row:

1,2017/01/03 10:14:26,002201001619,THREAT,url,1,2017/01/03 10:14:26,192.192.192.192,172.217.22.99,8.8.9.9,172.217.22.99,MyRule,mydomain\pippo.foo,,google-base,vsys1,LAN,UNTRUST,ae0,ae2,,2017/01/03 10:14:26,34041549,1,52109,80,21051,80,0x400000,tcp,alert,fonts.gstatic.com/s/oswald/v11/HqHm7BVC_nzzTui2lzQTDVtXRa8TVwTICgirnJhmVJw.woff2,(9999),computer-and-internet-info,informational,client-to-server,11243901,0x0,192.192.192.0-192.192.192.255,US,0,font/woff2,0,,,1,Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36,,,http://fonts.googleapis.com/css?family=Oswald:400,700,300,,,,0,0,0,0,0,,PALOALTO,

What went wrong ?
Do you have a workaround ?

5 Community Answers

Best Answer
Max Monterumisi Jan 05, 2017 04:21AM PST

RESOLVED!
Not export log files from WebGUI but export log files from Palo Alto CLI.
This trick give some advantages:
1) the problematic fields are delimitated by double quotas
2) you do not have the 65,535 row limit

An example of the command is:
scp export url start-time log equal 2017/01/04 @ 09: 00: 00 end-time equal 2017/01/04 @ 18: 00: 00 to root@192.168.2.117: /tmp/filone9-18.csv

After That, When you import the file on Vantage you do not have any problem.

Ficus.

View in conversation


Up 0 rated Down
Fastvue Jan 03, 2017 07:45AM PST FASTVUE Agent
Hey Massimiliano,

Thanks for getting in touch about this and sorry to hear about the issue. It does sound like a log parsing issue that you're experiencing.

Could you zip and upload your log to our upload site at http://www.fastvue.co/upload - we can then take a look at the parsing issues for you and hopefully get them resolved.

In the meantime, you could also try sending the Palo Alto Threat and Traffic logs via syslog to a syslog server that logs to text, then import the syslog text files. We have a free syslog server you can use for this at http://www.fastvue.co/syslog

For information on logging to syslog with Palo Alto, see our 'setting up a syslog server' section in :
http://webspy.com/blogs/reporting-on-palo-alto-firewalls-using-webspy-vantage/

The syslog format will hopefully not have the same parsing issues that you are encountering.

I look forward to hearing from you!

Cheers!
Scott
Up 0 rated Down
Max Monterumisi Jan 04, 2017 07:33AM PST
I guess one of possible problem is inside the csv.
The fields are separated by a comma, but the text isn't qualified with quotes or other character.
Then it's possible find inside some field the comma character and this populate the WebSpy DB with a wrong value.

For example this is a valid User Agent value:
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36

Other problem is: quote, double quote and other text qulified charter are used inside the URL, referer and other filed....
Up 0 rated Down
Fastvue Jan 04, 2017 07:42AM PST FASTVUE Agent
Hey Massimiliano,

That's right. We try to work around these issues as best we can, but sometimes issues do occur.

Thanks for uploading your log file. Allow us some time to investigate and resolve the parsing issues.

As mentioned, the syslog format (when collected with Fastvue Syslog - http://www.fastvue.co/syslog) does not have these issues. It's also a better solution for automating log imports as you don't need to manually export them from the Palo Alto box each time you need a report.

Just create a storage that points at the folder where your syslog text logs are created, and create a task for 1am each day that imports new hits to your existing storage. You can also schedule reports etc as part of the task.

Is syslog an option for you?

Cheers!
Scott
Up 0 rated Down
Max Monterumisi Best Answer Jan 05, 2017 04:21AM PST
RESOLVED!
Not export log files from WebGUI but export log files from Palo Alto CLI.
This trick give some advantages:
1) the problematic fields are delimitated by double quotas
2) you do not have the 65,535 row limit

An example of the command is:
scp export url start-time log equal 2017/01/04 @ 09: 00: 00 end-time equal 2017/01/04 @ 18: 00: 00 to root@192.168.2.117: /tmp/filone9-18.csv

After That, When you import the file on Vantage you do not have any problem.

Ficus.
Up 0 rated Down
Fastvue Jan 05, 2017 04:22AM PST FASTVUE Agent
Hey Max,

That's great news. Thanks for sharing your resolution to the Palo Alto manual log export issue.

Let us know if you have any other questions or issues!

Cheers!
Scott

Post Your Public Answer

Your name (required)
Your email address (required)
Answer (required)

Contact Us

support@fastvue.co
http://assets2.desk.com/
false
desk
Loading
seconds ago
a minute ago
minutes ago
an hour ago
hours ago
a day ago
days ago
about
false
Invalid characters found
/customer/en/portal/articles/autocomplete