+34 671518985 info@primecubeit.com

If we have a disk space limitation, it is important to monitor the size of the files/logs generated by Kafka and Postgres.

From Kafka side there are 3 logs that its size needs to be monitored if we have limited disk size:

  • Topic logs
    • The Topic log files are the data saved in Kafka Server and can be consumed by Kafka Consumer. The default path to store all the logs is ../MicroStrategy/install/MessagingServices/Kafka/tmp/kafka-logs. This folder size needs to be monitorized.
    • Log retention:

The below related settings can be modified in file ../MicroStrategy/install/MessagingServices/Kafka/kafka_2.12-2.2.0/config/server.properties”. 

Timeout: The minimum age of a log file to be eligible for deletion due to age. The default value is 168 hours, “log.retention.hours=168”. 

Please note that since the Diagnostics log configuration in the Intelligence Server is synchronized with Kafka Server at a frequency of 23 hours, the retention time should be set to be longer than 23 hours.

Size: Segments are pruned from the log unless the remaining segments drop below log.retention.bytes. Functions independently of log.retention.hours. The default value is 1073741824 bytes (1GB), “log.retention.bytes=1073741824”

Backup log size: The maximum size of a log segment file. When this size is reached, a new log segment will be created. The default value is 1073741824 bytes (1GB), “log.segment.bytes=1073741824”.

The administration of topic logs is a common concern.

  • Kafka broker log

    Besides the topic log files, there is another log direction for broker, controller and state-change. By default, this kind of log files will keep growing over time and will not be purged. The log files location is ../MicroStrategy/install/MessagingServices/Kafka/kafka_2.12-2.2.0/logs

Administrator can modify the configuration file ../MicroStrategy/install/MessagingServices/Kafka/kafka_2.12-2.2.0/config/log4j.properties to control the log files number and size. To do this we need to add two lines to the file, for example, the maximum number of server.log file will be 5, and each file size is 100KB.

log4j.appender.kafkaAppender.MaxFileSize=100KB

log4j.appender.kafkaAppender.MaxBackupIndex=5

  • Zookeeper Snapshot and Transition log

    Administrator can configure Zookeeper related settings in file ../MicroStrategy/install/MessagingServices/Kafka/kafka_2.12-2.2.0/config/zookeeper.properties.

By default, it includes the minimum configuration keywords that must be defined in the configuration file:

    • dataDir, the default value is ../MicroStrategy/install/MessagingServices/Kafka/tmp/zookeeper
    • Client port

For advanced setting please refer to https://zookeeper.apache.org/doc/r3.4.2/zookeeperAdmin.html#sc_advancedConfiguration.

A ZooKeeper server will not remove old snapshots and log files when using the default configuration. Thus, user can find the disk usage of folder “dataDir” continually increasing. Following advanced settings can be used to control the log directory’s size.

    • autopurge.snapRetainCount

When enabled, ZooKeeper auto purge feature retains the autopurge.snapRetainCount most recent snapshots and the corresponding transaction logs in the dataDir and dataLogDir respectively and deletes the rest. Defaults to 3. Minimum value is 3.

    • autopurge.purgeInterval

The time interval in hours for which the purge task has to be triggered. Set to a positive integer (1 and above) to enable the auto purging. This setting’s default value is 0.

There are two settings to limit the zookeeper transaction log size:

    • snapCount: This setting controls the number of transactions in one Zookeeper transaction log. By default, the Zookeeper transaction log switch will happen after “snapCount(100000)” transaction happens. We can reduce the size to smaller value, such as half – 50000.
    • preAllocSize: The setting limits the block size of each Zookeeper transaction log. The default value is 64MB, we can set it to 1MB.

One example is below. Please note that these settings are static ones, user need to restart Zookeeper to make them take effect. This values need to be added to the zookeeper config file:

autopurge.snapRetainCount=3
autopurge.purgeInterval=24
snapCount=5000
preAllocSize=1000

  •  
  • Postgres monitoring

Size of the Postgres can be monitored by use of the proper PA Dossier called “Database Capacity planning”.

It gives you an option to monitor both MicroStrategy repositories: PA and Collaboration Server.

To collect above statistics, the process needs to be started:

Capacity planning statistics collection is turned on by default after installation. The service is controlled with a batch file which accepts start | stop | restart | status as parameters.

From the command prompt, go to:

../MicroStrategy/install/Repository/repository-administration/bin

Call the following batch file with the applicable parameter:

.\\mstr-repo-dba-operations.sh start | stop | restart | status

To change the time of statistics collection, see KB483944.

    • Vacuuming

Whenever rows in a PostgreSQL table are updated or deleted, dead rows are left behind. Vacuum gets rid of them so that space can be reused. If a table doesn’t get vacuumed, it will get bloated, which wastes disk space and slows down sequential table scans. If the Vacuum option (3) is selected, the utility computes bloat for all databases and generates a report to the user.

Database Administration Tool

To launch the Database Administration Tool:

      • Open the command prompt and navigate to the repository administration

bin folder: ../MicroStrategy/install/Repository/repository-administration/bin

      • Run the file mstr-repo-ondemand-dba-operations.sh.

This will open the DBAOperations utility:

      • It shows you how much waste there in each repository and gives you option which database you want to clean:
      • Full vacuum requires consumer to be stopped and it is better when it is performed out of the work window:

      • Normal vacuum is done without any extra work:

      • There are more specific options when you vacuum individual DB:

      • As before, the full vacuum option requires consumer to be stopped:

Hopefully this PA series was useful and showed how to install, troubleshoot and monitor this product in this environment. Please leave a comment if there is anything in particular that you would like to read about in the MicroStrategy world.

References:

Share This