Overview
Each service emits a number of log statements when certain events within that service occur. When these events are logged they contain the relevant amount of written data and values to help you understand and trace that log but they also contain an Event ID as part of the log statement. This Event ID allows you to find a series of log statements or use queries to help better understand what events have occurred within a service.
You can collect and centralise the logs of Glasswall Halo using a log aggregation and monitoring tool, such as ELK Stack, Splunk, or Datadog. These tools provide powerful search and visualisation capabilities, allowing you to query logs based on Event IDs, timestamps, and other attributes.
Log levels
Log levels indicate the severity or importance of a log entry. This will help you prioritise and filter log messages. Here's how to understand and utilise log levels effectively:
Log level hierarchy
Log levels typically follow a hierarchy, with each level indicating increasing severity. Common log levels include:
- INFORMATION: Provides general information about the application's operation.
- WARNING: Highlights potential issues or unexpected behaviour that may not be critical.
- ERROR: Indicates errors that require immediate attention.
- CRITICAL: Reserved for the most severe issues that may lead to application failure.
Consumer actions
When you consume the log data you can take specific actions based on log levels:
- INFO: These logs are valuable for monitoring the regular operation of an application. Operations teams may use them to ensure the system is running smoothly.
- WARNING: Potential issues or irregularities are highlighted. Operations teams can set up alerts to proactively address these.
- ERROR: Immediate attention is needed. Operations teams should investigate and address errors promptly to prevent service disruptions.
- CRITICAL: Indicates a severe issue that requires immediate action. Operations teams should be alerted, and incident response processes should be initiated.
The Events omitted by Glasswall Halo are as follows:
Global event IDs
| Event |
ID |
Description |
Log Level |
| MongoDB-HealthCheckFailed |
100100 |
When the MongoDB Database health check fails |
Error |
| MongoDB-DocumentsRetrieved |
100101 |
When a query to the MongoDB database successfully returns documents |
Information |
| MongoDB-DocumentInserted |
100102 |
When documents are successfully inserted into the MongoDB database |
Information |
| MongoDB-DocumentsDeleted |
100103 |
When documents are successfully deleted from the MongoDB database |
Information |
| MongoDB-SumOfCalculated |
100104 |
When a SumOf query is successfully executed against the MongoDB database |
Information |
| MongoDB-DocumentsUpdated |
100105 |
When documents are successfully updated in the MongoDB database |
Information |
| MongoDB-DocumentsMatched |
100106 |
When an Exists query is successfully executed against the MongoDB database |
Information |
| CosmosDB-HealthCheckFailed |
100150 |
When the Cosmos Database health check fails |
Error |
| CosmosDB-ItemsRetrieved |
100151 |
When a query to the Cosmos database successfully returns items |
Information |
| CosmosDB-ItemInserted |
100152 |
When items are successfully inserted into the Cosmos database |
Information |
| CosmosDB-ItemsDeleted |
100153 |
When items are successfully deleted from the Cosmos database |
Information |
| CosmosDB-SumOfCalculated |
100154 |
When a SumOf query is successfully executed against the Cosmos database |
Information |
| CosmosDB-ItemsUpdated |
100155 |
When items are successfully updated in the Cosmos database |
Information |
| CosmosDB-ItemsMatched |
100156 |
When an Exists query is successfully executed against the Cosmos database |
Information |
Sync API
| Event |
ID |
Description |
Log Level |
| ReceivedAuthResponse |
5 |
When the API received the response from the Engine service |
Information |
| SuscribedToQueue |
6 |
When the API he subscribed to the response queue |
Information |
| CreatedQueue |
7 |
When the API creates the correct response queue |
Information |
| SentMessage |
8 |
When the API successfully submits a message to the Rabbit MQ |
Information |
| FileWritten |
9 |
When the API successfully writes a file to the file storage |
Information |
| FileRead |
10 |
When the API successfully reads the rebuilt file or analysis from storage |
Information |
| MessagePollerStarted |
11 |
When the rate limiting polling has started |
Information |
| IncorrectResponseMessage |
12 |
When the API received a response from the Engine Service but there is no task waiting to return the response to the user |
Warning |
| OperationFailed |
13 |
When an operation within the API fails |
Warning |
| RetryingPolicy |
14 |
When an operation retries based on the defined retry policy |
Warning |
| FailedRebuildArchiveEntry |
15 |
When the API fails to rebuild the clean files sections of an archive response |
Warning |
| FailedAnalysisArchiveEntry |
16 |
When the API fails to rebuild the analysis section of an archive response |
Warning |
| RateLimited |
17 |
When a request is not allowed due to rate limiting being reached |
Warning |
| ErrorHandlingRPCResponse |
18 |
When the API fails to handle a response message from the engine |
Error |
| FailedToSendTransactionEvent |
19 |
When the API fails to send transaction events used for dashboards |
Error |
| RequestFailed |
20 |
When the request to the API fails for a known reason |
Error |
| UnknownRequestFailure |
21 |
When the request to the API fails for an unknown reason |
Error |
| PipelineStarted |
22 |
When an action within the API begins |
Information |
| PipelineCompleted |
23 |
When an action within the API has completed |
Information |
| StartError |
24 |
When the API fails to start up |
Information |
| StorageNotFound |
25 |
When the API is unable to find the mounted storage location |
Information |
Engine
| Event |
ID |
Description |
Log Level |
| MetricsOutput |
2 |
When the Engine service logs metrics to stdout |
Information |
| SubscribedToQueue |
3 |
When the Engine service has subscribed to the request queue |
Information |
| CreatedQueue |
4 |
When the Engine service has created the request and logging queue |
Information |
| SentMessage |
5 |
When the Engine service has sent a message to the response or report queue |
Information |
| CompletedMessage |
6 |
When the Engine service has completed the request message from the API |
Information |
| RejectedMessage |
7 |
When the Engine service rejects the request message from the API |
RejectedMessage |
| GetStream |
8 |
When the Engine service gets stream of a file stored on the mounted disk |
Information |
| WrittenFile |
9 |
When the Engine service has written the rebuilt file to the mounted storage |
Information |
| ServiceStopping |
10 |
When the Engine service stops running |
Information |
| MessageReceivedDuringShutdown |
11 |
When the Engine service receives a message during its shutdown process |
Information |
| HandlingMessage |
12 |
When the Engine service begins to handle a request |
Information |
| WriteAnalysisFailed |
14 |
When the Engine service fails to write the analysis report to the mounted storage |
Warning |
| WriteCleanFileFailed |
15 |
When the Engine service fails to write the clean file to the mounted storage |
Warning |
| RetryingPolicy |
16 |
When the Engine service retries an operation based on a defined retry policy |
Warning |
| EngineVersionFailure |
17 |
When the Engine service fails to get the version from the embedded engine |
Warning |
| RabbitMqHealthFailure |
18 |
When the Rabbit MQ health check fails |
Warning |
| ErrorHandlingMessage |
20 |
When the Engine service fails to handle a request message |
Error |
| MessageError |
21 |
When the Engine Service has an issue processing a message from the request queue |
Error |
| ServiceDisconnected |
22 |
When the Engine service disconnects |
Critical |
| StorageNotFound |
23 |
When the Engine service cannot find the mounted storage |
Information |
| StartError |
24 |
When the Engine service fails to start |
Information |
| TrackerActionFinished |
25 |
When the Engine service completed tracking an event within the service |
Warning |
| ServiceStarting |
26 |
When the Engine service starts up |
Information |
| StorageHealthFailure |
27 |
When the storage health check fails |
Warning |
| WriteExportFileFailed |
28 |
When the Engine service fails to write the export file to the mounted storage |
Warning |
| DetectingPathFileType |
2001 |
When DetectFileTypeFromFile is called |
Information |
| DetectingMemoryFileType |
2002 |
When DetectFileTypeFromMemory is called |
Information |
| NoLicenseSetAtSessionCreation |
2003 |
When no active license was found. In this case, no CDR occurs |
Critical |
| EngineLicenseError |
2004 |
An active license was found, but it is invalid or expired |
Critical |
Report Extractor
| Event |
ID |
Description |
Log Level |
| UnableToConnectToMessageBroker |
1 |
When the service is unable to connect to the Message Broker at start up |
Critical |
| UnableToConnectToMessageBrokerQueue |
2 |
When the service is unable to connect to the response queue at start up |
Critical |
| UnexpectedMessageBrokerException |
3 |
When the API has an unexpected error connecting to the Message Broker or queue at start up |
Critical |
| StorageHealthFailure |
4 |
When the storage health check fails |
Error |
| ReadAnalysisReport |
5 |
When the Report extractor has read the analysis report from the mounted drive |
Information |
| FailedToReadAnalysisReport |
6 |
When the Report extractor fails to read the analysis report from the mounted drive |
Error |
API Access
| Event |
ID |
Description |
Log Level |
| AnonymousAuth |
1 |
When a request is allowed via anonymous auth |
Information |
| MissingAuthHeader |
2 |
When a request is missing the required auth header |
Information |
| ValidOrganisation |
3 |
When a request is made with the correct auth org ID |
Information |
| InvalidAuthHeader |
4 |
When a request is made with invalid auth header |
Information |
| ServiceFailedToStart |
5 |
When the service fails to start |
Critical |
| UpstreamResponse |
6 |
When an upstream service returns a response |
Information |
| UpstreamRequest |
7 |
When a request to an upstream service is made |
Information |
| RequestError |
8 |
When a request errors |
Error |
Portal Access
| Event |
ID |
Description |
Log Level |
| UpstreamResponse |
1 |
When an upstream service returns a response |
Information |
| UpstreamRequest |
2 |
When a request to an upstream service is made |
Information |
| StartError |
3 |
When the service fails to start |
Critical |
| RequestError |
4 |
When a request errors |
Error |
Async API
| Event |
ID |
Description |
Log Level |
| ServiceStarting |
1 |
When the API starts up |
Information |
| ServiceStopping |
2 |
When the shutdown process is called for Async API |
Information |
| MessageReceivedDuringShutdown |
3 |
When the API receives a message during its shutdown process |
Warning |
| HandlingMessage |
4 |
When the API is handling a message from the queue |
Information |
| HandledMessage |
5 |
When the API has successfully handled a message from the queue |
Information |
| MessageError |
6 |
When the API has an issue processing a message from the queue |
Error |
| ServiceDisconnected |
7 |
When the API disconnects from the message broker |
Critical |
| SubscribedToQueue |
8 |
When the API subscribed to receive message from the queue |
Information |
| SentMessage |
9 |
When the API has sent a message for processing |
Information |
| FileWritten |
10 |
When the API successfully writes a file to the file storage |
Information |
| FileRead |
11 |
When the API successfully reads the rebuilt file or analysis from storage |
Information |
| IncorrectStatusStored |
12 |
When the status for a stored item is not the expected status |
Warning |
| OperationFailed |
13 |
When an operation within the API fails |
Critical |
| RetryingPolicy |
14 |
When an operation retries based on the defined retry policy |
Warning |
| RequestFailed |
16 |
When the request to the API fails for a known reason |
Error |
| UnknownRequestFailure |
17 |
When the request to the API fails for an unknown reason |
Error |
| PipelineStarted |
18 |
When an action within the API begins |
Information |
| PipelineCompleted |
19 |
When an action within in the API has completed |
Information |
| StartError |
20 |
When the API fails to start up |
Information |
| StorageNotFound |
21 |
When the API is unable to find the mounted storage location |
Information |
| HashingError |
22 |
When the API is unable to generate hashes for the file |
Error |
| DeleteFailed |
23 |
When the API is unable to delete the file from the mounted storage location |
Error |
| UnableToConnectToMessageBroker |
24 |
When the API is unable to connect to the Message Broker at start up |
Critical |
| UnableToConnectToMessageBrokerQueue |
25 |
When the API is unable to connect to the response queue at start up |
Critical |
| UnexpectedMessageBrokerException |
26 |
When the API has an unexpected error connecting to the Message Broker or queue at start up |
Critical |
| DatabaseHealthFailure |
27 |
When the API's database health check fails to connect to the database |
Warning |
| MessageBrokerHealthFailure |
28 |
When the API's message broker health check fails to connect to the message broker |
Warning |
| DatabaseFailure |
29 |
When the API is unable to complete an operation on the database after maximum number of retries |
Error |
| DatabaseItemRetrieved |
30 |
When the API has performed a retrieval on an item in the database |
Information |
| DatabaseItemStored |
31 |
When the API has stored an item in the database |
Information |
| DatabaseItemUpdated |
32 |
When the API has updated an item in the database |
Information |
| HandleSingleFileResponse |
33 |
When the API has received a response relating to a single file |
Warning |
| HandleArchiveContentResponse |
34 |
When the API has received a response relating to an archive content item |
Warning |
Tally Accumulator
| Event |
ID |
Description |
Log Level |
| UnableToConnectToMessageBroker |
1 |
When the API is unable to connect to the Message Broker at start up |
Critical |
| UnableToConnectToMessageBrokerQueue |
2 |
When the API is unable to connect to the response queue at start up |
Critical |
| UnexpectedMessageBrokerException |
3 |
When the API has an unexpected error connecting to the Message Broker or queue at start up |
Critical |
| StartError |
4 |
When the API fails to start up |
Critical |
| ProcessTransactionEventStart |
5 |
When a supported transaction is received and processing has started |
Information |
| ProcessTransactionEventEnd |
6 |
When a supported transaction is received and processing has finished successfully |
Information |
| LicenseTransactionStoreUpsertStart |
7 |
When an upsert is about to happen against the License Transaction Store |
Information |
| LicenseTransactionStoreUpsertSuccess |
8 |
When an upsert against the License Transaction Store has finished successfully |
Information |
| TallyStoreUpsertStart |
9 |
When an upsert is about to happen against the Tally Store |
Information |
| TallyStoreUpsertComplete |
10 |
When an upsert against the Tally Store is successful |
Information |
| TallyStoreGetUsageStart |
11 |
When license usage is about to be retrieved from the Tally Store |
Information |
| TallyStoreGetUsageSuccess |
12 |
When license usage is retrieved from the Tally Store successfully |
Information |
| UnexpectedNegativeFileSize |
13 |
When a license transaction event for a file has a negative file size |
Warning |
| UnexpectedRequestException |
14 |
When an unexpected exception is raised by the service that will result in a 500 status code for a given API request |
Error |
| UnsupportedEntitlementLimitName |
15 |
When the supplied filter for the usage endpoint is not a valid entitlement limit |
Warning |
| DatabaseHealthFailure |
16 |
When the service health checks fail to contact the database instances |
Warning |
| ErroredFileReceived |
17 |
When an error occurs upstream. This will result in the tally not being incremented |
Warning |
Metrics Collation (Deprecated)
As of version 2.6.2 of Halo, the Metrics Collation service is no longer required.
| Event |
ID |
Description |
Log Level |
| UnableToConnectToMessageBroker |
1 |
When the service is unable to connect to the Message Broker at start up |
Critical |
| UnableToConnectToMessageBrokerQueue |
2 |
When the service is unable to connect to the response queue at start up |
Critical |
| UnexpectedMessageBrokerException |
3 |
When the service has an unexpected error connecting to the Message Broker or queue at start up |
Critical |
| StorageHealthFailure |
4 |
When the storage health check fails; this is usually a problem with database connectivity |
Error |
| WritingEventToMongoStorage |
5 |
When an incoming event is about to be persisted in the database |
Information |
| MongoWriteError |
7 |
When an incoming event fails to be persisted in the database |
Error |
| JsonExceptionError |
8 |
When the message listener fails to deserialize an event |
Error |
| UnknownFailedToProcess |
10 |
When an unhandled exception occurs while processing an incoming event |
Critical |
| FailedToImportTraceData |
11 |
When the trace provider fails to import trace data from an incoming event header |
Warning |
| StorageHealthCheckStartUp |
12 |
When the service performs a storage health check at startup |
Information |
Metrics Projection
| Event |
ID |
Description |
Log Level |
| StartError |
1 |
When the service fails to start up |
Critical |
| StorageHealthFailure |
2 |
When the storage health check fails; this is usually a problem with database connectivity |
Warning |
| ChangeProcessorStart |
5 |
When the change processor is about to setup a watcher for database changes |
Information |
| ChangeProcessorStarted |
6 |
When the change processor has successfully setup a watcher for database changes |
Information |
| ChangeProcessorStopping |
7 |
When the change processor is about to stop watching for database changes |
Information |
| ChangeProcessorStopped |
8 |
When the change processor has successfully stopped watching for database changes |
Information |
| ChangeProcessorStopFailure |
9 |
When the change processor fails to stop watching for database changes, i.e. it was not watching for changes |
Warning |
| ChangeProcessorBatchStart |
10 |
When the change processor has recieved a batch of database changes and is about to start procoessing them |
Information |
| ChangeProcessorBatchEnd |
11 |
When the change processor has finished processing the current batch of database changes |
Information |
| ChangeProcessorUnhandledException |
13 |
When the change processor encounters a problem while processing a change; this may cause inconsistencies in data |
Error |
| TransactionEventProjected |
14 |
When an event (change) has been successfully mapped to a projection |
Debug |
| ChangeHandlerStarted |
15 |
When a change handler is about to start processing changes after being called upon by the change processor |
Information |
| ChangeHandlerFinished |
16 |
When the change handler has finished processing changes |
Information |
| ChangeHandlerFailure |
17 |
When the change handler has failed to process changes |
Critical |
| ChangeProcessorStartFailure |
19 |
When the changle processor fails to start watching for database changes |
Error |
| ChangeProcessorWatchStart |
20 |
When the change processor has started actively watching for database changes (after setting up a watcher) |
Information |
| DaywiseSummaryIncremented |
21 |
When a day-wise summary has been successfully incremented based on a completed file |
Information |
| DaywiseSummaryNotIncremented |
22 |
When a day-wise summary will not be incremented based on a (usually incomplete) file |
Debug |
| RetryingPolicy |
23 |
When an operation retries based on the defined retry policy |
Warning |
| MaterializedViewPersisted |
24 |
When a materialized view (projection) has been persisted in the database after being projected upon |
Information |
License Management
| Event |
ID |
Description |
Log Level |
| StartError |
1 |
When the service fails to start up |
Critical |
| LicenseProcessingError |
2 |
When a request fails for an unforeseen reason |
Error |
| DeleteLicenseStart |
3 |
When the service begins deleting the current active license |
Information |
| DeleteLicenseEnd |
4 |
When the service has finished deleting the current active license |
Information |
| UpdateLicenseStart |
5 |
When the service begins updating the current active license |
Information |
| UpdateLicenseEnd |
6 |
When the service has finished updating the current active license |
Information |
| GetLicenseStart |
7 |
When the service begins retrieving the current active license |
Information |
| GetLicenseEnd |
8 |
When the service has finished retrieving the current active license |
Information |
| LicenseNotFound |
9 |
When the current active license cannot be found during update or retrieval |
Error |
| LicenseExpired |
10 |
When the uploaded license has expired, causing the update to fail |
Error |
| FetchingLicenseUsageStart |
11 |
When the service begins retrieving the current active license and usage data |
Information |
| FetchingLicenseUsageEnd |
12 |
When the service has finished retrieving the current active license and usage data |
Information |
ICAP Server
| Event |
Id |
Description |
Log Level |
| BackgroundServiceStarting |
1 |
When the ICAP listener is starting |
Information |
| BackgroundServiceStarted |
2 |
When the ICAP listener has started |
Information |
| BackgroundServiceStopping |
3 |
When the service shuts down and the ICAP listener is stopping |
Information |
| BackgroundServiceStopped |
4 |
When the service shuts down and the ICAP listener has completed stopping |
Information |
| StartError |
5 |
When the service fails to start |
Critical |
| HaloCallFailure |
8 |
When a file cannot be built via the Halo API |
Warning |
| ProfileStoreMaintainerStarting |
9 |
When the background thread starts listening for ICAP profile changes to persist locally |
Information |
| ProfileStoreMaintainerStarted |
10 |
When the background thread listening for ICAP profile changes has started successfully |
Information |
| ProfileStoreMaintainerStopping |
11 |
When the service shuts down and the profile background thread stops listening for profile updates |
Information |
| ProfileStoreMaintainerStopped |
12 |
When the service shuts down and the profile background thread completes its shutdown process |
Information |
| ProfileStoreMaintainerHeartbeatError |
13 |
When profiles are queried and cannot be persisted due to an error |
Error |
| ProfileStoreMaintainerHeartbeatSuccess |
14 |
When profiles are queried successfully |
Information |
| ProfileUsageFailure |
15 |
When the profile specified in an ICAP request is not found |
Warning |
| ErrorHandled |
17 |
When the icap server finishes processing an unexpected exception by sending an error response to the icap client |
Error |
| UnlicensedRequest |
18 |
When an ICAP request is received for an unlicensed icap server |
Critical |
| RetryingPolicy |
19 |
When an action is being retried, for example a failed call into the Halo API |
Warning |
| FileBlocked |
20 |
When an ICAP request is processed containing a file that is blocked |
Information |
| FileProcessingStarted |
21 |
When starting to process a file within an icap request |
Information |
| ProfileFailedToFetch |
22 |
When a profile cannot be retrieved from the Policy API |
Error |
| RebuiltFileExceedsCacheSize |
23 |
When the ICAP request contains a file too large to be stored in the cache |
Warning |
| FileRetrievedFromCache |
24 |
When an ICAP request is processed and an associated rebuilt file is found in the cache |
Information |
| FileProcessingFinished |
25 |
When finished processing a file within an icap request |
Information |
| ClearingCache |
26 |
When running cache cleanup starts |
Information |
| CacheCleared |
27 |
When running cache cleanup finishes |
Information |
| ErrorClearingCache |
28 |
When running cache cleanup errors |
Error |
| FileBypassed |
29 |
When a file has been bypassed during modification by the ICAP policy in use |
Information |
| FileProcessed |
30 |
When a file has been processed during modification by the ICAP policy in use |
Information |
| FileErrored |
31 |
When an error occurs while attempting to process a file during modification |
Information |
| TlsDisabled |
32 |
When no valid certificates are provided to the ICAP server, the server stops listening on secure port 11344 and TLS is disabled. |
Warning |
| TlsEnabled |
33 |
The ICAP server has found a valid TLS certificate and is listening on the secure port 11344. |
Information |
| ServerStarting |
1000 |
When the ICAP server is starting |
Information |
| ServerStarted |
1001 |
When the ICAP server has started |
Information |
| ClientSocketError |
1002 |
When an unexpected exception occurs while listening to the icap client |
Error |
| UnsupportedIcapRequest |
1003 |
When the icap server receives a request that is not supported, for example the Method is not REQMOD, OPTIONS or RESPMOD |
Warning |
| ClientConnectionHandleError |
1004 |
When an error occurs when processing data sent to the server |
Error |
| SendFailure |
1005 |
When attempting to send a response to the icap client fails |
Warning |
| ServerListenerCancelled |
1006 |
When the ICAP listener thread is cancelled, for example on shutdown |
Warning |
| ErrorDuringSocketClose |
1007 |
When closing a connection raises an error, for example when the client has already disconnected |
Information |
| ClientReceiveError |
1008 |
When listening to data on the socket sent by the ICAP client and error occurs |
Warning |
| RequestProcessError |
1009 |
When an unhandled exception occurs during the processing of an ICAP request |
Error |