Storage Space Reduced to One-Tenth of MySQL — The Application of TDengine in Monitoring Scenario of Huolala Database

  1. Problems Encountered in the Development of Monitoring System
  • Using a distributed architecture, and it can support linear expansion
  • Data storage compression ratio is very high
  • Cluster mode supports multiple copies, no single point of failure
  • Powerful performance in stand-alone mode
  • Simple installation, deployment, and maintenance
  • SQL supports time dimension aggregation query
  • Client version supports RESTful API
  • Transform the data structure of monitoring data delivery
  • Transform the consumption program, consumption logic, reorganize data structure
  • TDengine, the monitoring data of each data storage is built separately and stored
  • TDengine, build a super table, take Redis as an example
  • Monitoring data consumption, since the newly added instance nodes are uncertain, for example, the node resources of Redis are automatically discovered and registered by the Agent and automatically collect monitoring indicators. At this time, the data is written and it is not certain whether a sub-table exists, so TDengine’s automatic table creation syntax is required and used to create a sub-table that does not exist. If the sub-table already exists, a new table will not be created and only data will be written.
  • For drawing display, the drawing program does not have a client version driver installed, and directly uses the RESTful API provided by TDengine, and uses HTTP to query data. The time dimension aggregation function INTERVAL provided by TDengine is widely used in SQL query. For long-term data query, only a reasonable selection of aggregation interval is required, and the response is basically in milliseconds, which ensures the smooth and stable front-end drawing.
  • Since TDengine does not support too complex SQL query syntax, it is necessary to be extremely thoughtful when designing super table tags. Currently, only tag fields support GROUP BY query of function operation results, and common fields are not supported.
  • The value of the subtable tag can be changed, but the value of the same subtable tag is unique. It is recommended to use the combined value of the subtable tag to generate the subtable name.

Unlisted

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
TDengine

TDengine

Open-source, cloud-native time-series database optimized for IoT. See more at https://tdengine.com or https://github.com/taosdata/TDengine