Prometheus Monitoring Exporter

prometheus-grafana-loki-logging-service

prometheus-alert-service"}[$__interval]))`, `{service=~"prometheus-grafana-loki-logging-service", env=~"$Kafka_Data_Source_Env"}` ```bash pip install python-logging-loki import logging import logging_loki logging_loki.emitter.LokiEmitter.level_tag = "level" # assign to a variable named handler handler = logging_loki.LokiHandler( url="http://loki:3100/loki/api/v1/push", version="1", ) # create a new logger instance, name it whatever you want logger = logging.getLogger("my-logger") logger.addHandler(handler) # now use the logging object's functions as you normally would logger.error( "Something bad happened", extra={"tags": {"service": "my-service"}}, ) logger.warning( "Something bad happened but we can keep going", extra={"tags": {"service": "my-service"}}, ) # extra={"tags": {"service": "my-service", "one": "more thing"}} ``` - Grafana Promtail Setup ```bash wget https://github.com/grafana/loki/releases/download/v2.8.0/promtail-linux-amd64.zip unzip promtail-linux-amd64.zip nohup ./promtail-linux-amd64 -config.file=promtail-config.yaml 2>&1 & - How to use : https://kbcoding.tistory.com/122 # -- server: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: /tmp/positions.yaml clients: - url: http://loki:3100/loki/api/v1/push scrape_configs: - job_name: logging static_configs: - targets: - localhost labels: job: logging __path__: /logs/*.log # -- ``` - Promtail Format  ### Installation ```bash pip install prometheus-client ``` ### POST JSON - POST Json payload command ```bash curl -X 'POST' \ 'https://localhost/triggers' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -H 'Authorization: Basic test=' \ -d '{ "recipients": "TEST app", "summary": "[TEST] Prometheus Monitoring Alert", "description": "The alert is a message for testing", "priority": "MEDIUM" }' ``` ### Custom Promethues Exporter - Expose my metrics for dev kafka cluster to http://localhost:9115 - Expose ES cluster/Kafka/Kibana/Logstash metrics by using this exporter based on ES API/socket library - EXpose DB records as metrics from `DB-Interface-Export API` with HTTP POST Method - Interface with `DB Interface Export` (https://github.com/euiyounghwang/DB-Interface-Export) by using FastAPI Framework to get the records as metrics from the specific databse ```bash # HELP python_gc_objects_collected_total Objects collected during gc # TYPE python_gc_objects_collected_total counter python_gc_objects_collected_total{generation="0"} 302.0 python_gc_objects_collected_total{generation="1"} 342.0 python_gc_objects_collected_total{generation="2"} 0.0 ... kafka_health_metric{server_job="localhost"} 3.0 # HELP kafka_connect_nodes_metric Metrics scraped from localhost # TYPE kafka_connect_nodes_metric gauge kafka_connect_nodes_metric{server_job="localhost"} 3.0 # HELP kafka_connect_listeners_metric Metrics scraped from localhost # TYPE kafka_connect_listeners_metric gauge kafka_connect_listeners_metric{host="localhost",name="test_jdbc",running="RUNNING",server_job="localhost"} 1.0 # HELP zookeeper_health_metric Metrics scraped from localhost # TYPE zookeeper_health_metric gauge zookeeper_health_metric{server_job="localhost"} 3.0 # HELP kibana_health_metric Metrics scraped from localhost # TYPE kibana_health_metric gauge kibana_health_metric{server_job="localhost"} 1.0 # HELP logstash_health_metric Metrics scraped from localhost # TYPE logstash_health_metric gauge logstash_health_metric{server_job="localhost"} 1.0 ... ``` ### Run Custom Promethues Exporter - Run this command : $ `python ./standalone-es-service-export.py` or `./standalone-export-run.sh` ```bash #-- HTTP Server/Client Based # HTTP Server $ ./es-service-all-server-export-run.sh status/start/stop or ./server-export-run.sh Server started. # Client export $ ./es-service-all-client-export-run.sh status/start/stop or ./client-export-run.sh # Client only ./standalone-es-service-export.sh status/start/stop ... [2024-05-20 20:44:06] [INFO] [prometheus_client_export] [work] http://localhost:9999/health?kafka_url=localhost:9092,localhost:9092,localhost:9092&es_url=localhost:9200,localhost:9200,localhost:9200,localhost:9200&kibana_url=localhost:5601&logstash_url=process [2024-05-20 20:44:06] [INFO] [prometheus_client_export] [get_metrics_all_envs] 200 [2024-05-20 20:44:06] [INFO] [prometheus_client_export] [get_metrics_all_envs] <Response [200]> [2024-05-20 20:44:06] [INFO] [prometheus_client_export] [get_metrics_all_envs] {'kafka_url': {'localhost:9092': 'OK', 'GREEN_CNT': 3, 'localhost:9092': 'OK', 'localhost:9092': 'OK'}, 'es_url': {'localhost:9200': 'OK', 'GREEN_CNT': 4, 'localhost:9200': 'OK', 'localhost:9200': 'OK', 'localhost:9200': 'OK'}, 'kibana_url': {'localhost:5601': 'OK', 'GREEN_CNT': 1}, 'logstash_url': 1} ... ``` ### Create script with arguments with python - Run this command : euiyoung.hwang@US-5CD4021CL1-L MINGW64 ~/Git_Workspace/python-prometheus-export/create-script (master) $ `python ./create-script-by-hosts.py` ```bash euiyoung.hwang@US-5CD4021CL1-L MINGW64 ~/Git_Workspace/python-prometheus-export/create-script (master) $ python ./create-script-by-hosts.py 2024-05-31 19:58:16,573 : INFO : { "dev": { "kibana": "kibana:5601", "es_url": [ "es1:9200", "es2:9200", "es3:9200", "es4:9200" ], "kafka_url": [ "data1:9092", "data2:9092", "data3:9092" ], "kafka_connect_url": [ "data1:8083", "data2:8083", "data3:8083" ], "zookeeper_url": [ "data1:2181", "data2:2181", "data3:2181" ] }, "localhost": { "kafka_url": [ "data11:9092", "data21:9092", "data31:9092" ], "kafka_connect_url": [ "data11:8083", "data21:8083", "data31:8083" ], "zookeeper_url": [ "data11:2181", "data21:2181", "data31:2181" ], "kibana": "kibana1:5601", "es_url": [ "es11:9200", "es21:9200", "es31:9200", "es41:9200", "es51:9200" ] } } # dev ENV python ./standalone-es-service-export.py --interface http --db_http_host localhost:8002 --url jdbc:oracle:thin:bi"$"reporting/None --db_run false --kafka_url data1:9092,data2:9092,data3:9092 --kafka_connect_url data1:8083,data2:8083,data3:8083 --zookeeper_url data1:2181,data2:2181,data3:2181 --es_url es1:9200,es2:9200,es3:9200,es4:9200 --kibana_url kibana:5601 --interval 30 --sql "SELECT * FROM TEST*" # localhost ENV python ./standalone-es-service-export.py --interface http --db_http_host localhost:8002 --url jdbc:oracle:thin:bi"$"reporting/None --db_run false --kafka_url data11:9092,data21:9092,data31:9092 --kafka_connect_url data11:8083,data21:8083,data31:8083 --zookeeper_url data11:2181,data21:2181,data31:2181 --es_url es11:9200,es21:9200,es31:9200,es41:9200,es51:9200 --kibana_url kibana1:5601 --interval 30 --sql "SELECT * FROM TEST*" ``` ### Service Maintance - Kafka Service ```bash /apps/kafka_2.11-0.11.0.0/bin/kafka-topics.sh --describe --zookeeper localhost.am.co.gxo.com:2181,localhost.am.co.:.com:2181,localhost.am.co.gxo.com:2181 --topic ELASTIC_PIPELINE_QUEUE ./test_jdbc.json { "name": "test_jdbc", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "timestamp.column.name": "ADDTS", "tasks.max": "1", "transforms.InsertKey.fields": "SRC_TBL_KEY", "transforms": "InsertKey,ExtractId", "batch.max.rows": "5000", "timestamp.delay.interval.ms": "1000", "table.types": "TABLE", "transforms.ExtractId.field": "SRC_TBL_KEY", "query": "select * from (SELECT * FROM test_schema.test_VM where status = 'N')", "mode": "timestamp", "topic.prefix": "TEST_QUEUE", "poll.interval.ms": "1000", "schema.pattern":"bi$reporting", "connection.backoff.ms":"60000", "connection.attempts":"10", "connection.pool.size": "5", "connection.url": "jdbc:oracle:thin:test/testw@localhost:1234/testdb", "transforms.InsertKey.type": "org.apache.kafka.connect.transforms.ValueToKey", "transforms.ExtractId.type": "org.apache.kafka.connect.transforms.ExtractField$Key" } } curl -XGET 'localhost:8083/connectors/test_jdbc/status'