Adv Troubleshooting

AWS

Important to know

Cloud Bot remediation configuration takes up to 20 minutes to take effect
AWS logs are delayed, ~5 minutes is normal

Troubleshooting

90% of issues are resolved by going over the setup instructions and confirming each step is correctly configured. If that didn’t help, see below.

Review Cloud Bots deployment in AWS

Review stack (default name is dome9CloudBots)

check created date
check last updated date
- should be after last updated for https://github.com/dome9/cloud-bots/tree/master
- we currently don’t do releases – update to latest
- Update by selecting “Update” on the stack and going through the steps
  - This will overwrite custom CloudBots
note inputArn
note outputArn
- Customer can subscribe to outputArn with email for troubleshooting / monitoring
note LambdaArn

Review Lambda

environment variables
- sendLogs should be false (currently broken, pending DFT)
- single / multi deployment? should usually be “single” unless intended to be multi
code
- check last updated
- if after stack update: someone made code changes on the fly.

Review CloudGuard configuration

Posture Management > Continuous Posture

continuous compliance configuration
review notification configuration
- correct inputArn?
- full JSON entity?
review remediation configuration
- correct bot for rule?
  - review rule and bot description
  - valid parameters? compare to bot documentation
- correct scope?
Trigger “Send all alerts” to the SNS
Changes to remediation config are not reflected in real-time. It will take anywhere from 10-30 minutes for an update to be reflected in the messages sent to SNS.

Review Lambda logs

Check the latest invocations in CloudWatch
- If any errors are present, review. Common errors are listed below.
- Code is easy to review on GitHub, the logs will tell you which file to look at.

Common Errors

Error: Provided region_name ‘N. Virginia’ doesn’t match a supported format.

The notification is configured with JSON - Basic entity instead of JSON - Full entity.

KeyError ‘Records’

The lambda code may be outdated. Update the stack to latest version

AttributeError: module ‘botocore.vendored.requests’ has no attribute ‘Session’

Ignore. sendLogs is broken. Set SEND_LOGS_TO_DOME9 to false to make it go away.

Important to know

Cloud Bot remediation configuration takes up to 20 minutes to take effect
AWS logs are delayed, ~5 minutes is normal

Troubleshooting

90% of issues are resolved by going over the setup instructions and confirming each step is correctly configured. If that didn’t help, see below.

Review Cloud Bots deployment in AWS

Review stack (default name is dome9CloudBots)

check created date
check last updated date
- should be after last updated for https://github.com/dome9/cloud-bots/tree/master
- we currently don’t do releases – update to latest
- Update by selecting “Update” on the stack and going through the steps
  - This will overwrite custom CloudBots
note inputArn
note outputArn
- Customer can subscribe to outputArn with email for troubleshooting / monitoring
note LambdaArn

Review Lambda

environment variables
- sendLogs should be false (currently broken, pending DFT)
- single / multi deployment? should usually be “single” unless intended to be multi
code
- check last updated
- if after stack update: someone made code changes on the fly.

Review CloudGuard configuration

Posture Management > Continuous Posture

continuous compliance configuration
review notification configuration
- correct inputArn?
- full JSON entity?
review remediation configuration
- correct bot for rule?
  - review rule and bot description
  - valid parameters? compare to bot documentation
- correct scope?
Trigger “Send all alerts” to the SNS
Changes to remediation config are not reflected in real-time. It will take anywhere from 10-30 minutes for an update to be reflected in the messages sent to SNS.

Review Lambda logs

Check the latest invocations in CloudWatch
- If any errors are present, review. Common errors are listed below.
- Code is easy to review on GitHub, the logs will tell you which file to look at.

Common Errors

Error: Provided region_name ‘N. Virginia’ doesn’t match a supported format.

The notification is configured with JSON - Basic entity instead of JSON - Full entity.

KeyError ‘Records’

The lambda code may be outdated. Update the stack to latest version

AttributeError: module ‘botocore.vendored.requests’ has no attribute ‘Session’

Ignore. sendLogs is broken. Set SEND_LOGS_TO_DOME9 to false to make it go away.

GCP

Azure

Important to know

Cloud Bot remediation configuration takes up to 20 minutes to take effect
Azure logs are delayed, ~5 minutes is normal

Troubleshooting

90% of issues are resolved by going over the setup instructions and confirming each step is correctly configured. If that didn’t help, see below.

Collecting information

Everything in this section should be collected as evidence in case further support is required.

In CloudGuard

Check Continuous Compliance setup in CloudGuard
Policy?
Notification?
Check Remediation settings in CloudGuard
Check Notification settings in CloudGuard
Basic auth?
Correct URL?
URL includes ?code=XXXAAZXA?
Full JSON entity

In Azure

Check Python logs
Function > Monitor > Invocations
Usually what you need to identify any issue from the output.
Check environment variables.
Function App > Settings > Configuration
Get App Insights logs.
Function App > Monitoring > Log stream
Will only have good logs once basic setup works and events are parsed.

Common issues

401 Unauthorized

By default, the function is deployed with a function key. This function key is not revealed on the command line after running func azure functionapp publish .

Either use the URL with function key or set to “anonymous” in dome9CloudBots/function.json then redeploy. If latter, make sure you have basic authentication configured, otherwise anyone can send commands to the Cloud Bots.

Double check username and password afterwards.

curl for testing
This will make sure that basic setup of the function was successful and authorization is correctly configured.

curl -X POST -i \
    --data '{
        "account": {
            "id": "Dummy"
        },
        "status": "Passed",
        "rule": {
            "complianceTags": ""
        },
        "remediationActions": []
    }' \
  https://username:password@function_name.azurewebsites.net/api/dome9CloudBots

Expected result

HTTP/1.1 200 OK Content-Type: text/plain; charset=utf-8 Date: Mon, 02 Oct 2023 20:01:15 GMT Server: Kestrel Transfer-Encoding: chunked Request-Context: appId=cid-v1:

{'Account id': 'Dummy', 'Finding key': 'N.A'}