Pushing to Prometheus

Prometheus is thoroughly a pull-based system, attempting to push to it instead will not likely end well for you. There is a lot of information and misinformation online on push vs. pull, but the short version is that overall it largely doesn't matter. There are a few cases where Prometheus's pull-based approach requires some additional effort (just as there are for push), which you may run into.


Pull depends on having something to scrape, which does not work well for batch or cron jobs. If the batch job is tied to a machine and the machine's lifecycle (i.e. when the machine goes away, you no longer care about the metrics from it) then you probably want to use the Node exporter's textfile collector. If the batch job is a service level batch job (roughly speaking, it has no instance label as it does work across to an entire job/service) then you should use the Pushgateway. The Pushgateway and textfile collector are not ways to generally perform push though, they are intended to deal with the results of batch jobs. If you decommission a batch job, don't forget to include a step to delete its data from the Pushgateway/textfile collector.


Certain network setups require clients to initiate the TCP connection rather than the Prometheus server. While it is recommended to run Prometheus on the same network as what it is monitoring with no intervening firewalls or load balancers, this isn't always possible. In such cases you can look at using proxy servers to allow access to the clients, or solutions such as PushProx. You may also find yourself needing service discovery in such a situation, if you do not have something already. Consul is one popular choice.