<SYSTEM>This is the abridged developer documentation for Tier0</SYSTEM>

# Installation

> Install Tier0 by edition.

Tier0 ships in three editions: * **Cloud** - Managed SaaS, no infrastructure required. * **Enterprise** - Private deployment for enterprise environments. * **Edge** - Open-source, runs locally with Docker. See [Choosing the Best Version](../choosing-the-best-product/) to find the edition that best fits your requirements. ## Installing Tier0 [Section titled “Installing Tier0”](#installing-tier0) * Cloud Apply for a trial account before exploring Tier0 Cloud. 1. Register your account at [Register Tier0](https://tier0.dev/signup). 2. Use the verification code sent to your email to verify and then log in with the registered account. 3. Create a workspace and start your trial. * Enterprise #### Requirements [Section titled “Requirements”](#requirements) * Standard Deployment * Minimum: 4 VMs * Required baseline per VM types: * App: 8C / 32G / 100G * Database: 4C / 8G / 500G * Witness: 2C / 4G / 40G * High Availability Deployment * Minimum: 6 VMs * Required baseline per VM types: * App (Swarm): 8C / 32G / 100G x 3 * Database: 4C / 8G / 500G x 2 * Witness: 2C / 4G / 40G #### Operations [Section titled “Operations”](#operations) 1. Get the package from your vendor, and verify it. 
     ```bash
     sha256sum -c tier0-enterprise-offline-standard-YYYYmmddTHHMMSS.tar.gz.sha256
     ```
 2. Decompress the package. 
     ```bash
     tar -xzf tier0-enterprise-offline-standard-YYYYmmddTHHMMSS.tar.gz
     cd tier0-enterprise-offline-standard-YYYYmmddTHHMMSS
     ```
 3. Edit `install.env` based on the deployment details, then start the deployment. 
     ```bash
     vi install.env
     bash install.sh
     ```
 #### Common Maintenance Operations [Section titled “Common Maintenance Operations”](#common-maintenance-operations) * Check the deployment environment without installing Enterprise. 
    ```bash
    ./install.sh --check-only
    ```
 * Clear all nodes after a failed deployment. 
    ```bash
    ./full-cleanup.sh --env install.env --yes
    ```
 What difference it makes This command wipes all Tier0 runtime data and database backups while keeping the VMs, operating system, and installed Docker software. * A deployment report is generated after installation. 
    ```text
    runtime/install-reports/<timestamp>/deployment-report.md
    ```
 What you should do when **FAIL** appears in the report Do not deliver the project if the report contains any **FAIL** item. Contact support first. #### Maintenance References [Section titled “Maintenance References”](#maintenance-references) * [Standard Port List](../../reference/standard-port-list/) * [Maintenance and Troubleshooting Guide](../../reference/operations-runbook/) * [HA Boundaries Instructions](../../reference/sla-and-ha-boundaries/) * Edge #### Requirements [Section titled “Requirements”](#requirements-1) * Minimum 4-core CPU, 8 GB RAM, 100 GB disk * Ubuntu 24.04 or Windows 10/11 with Docker and Git #### Operations [Section titled “Operations”](#operations-1) 1. Clone the Tier0-Edge repository from GitHub to your server and access the project. 
     ```bash
     git clone https://github.com/FREEZONEX/Tier0-Edge
     cd Tier0-Edge/deploy
     ```
 2. Configure the parameters listed in the table below in `.env.default` according to your environment, and save the changes. 
     ```bash
     cp .env.default .env
     vi .env
     :wq!
     ```
 3. Run the installation script and check the installation status. 
     ```bash
     bash bin/install.sh
     bash bin/compose.sh ps # Check the status of the containers
     bash bin/compose.sh logs -f backend # Check the logs of the backend container
     curl -fsS http://127.0.0.1:8088/readyz # Check the readiness of the backend service
     ```
 Caution Do not edit `deploy/.env.runtime`; it contains generated secrets and resolved runtime values. 4. Access Tier0 with the set endpoint and the default admin account: * Username: `tier0` * Password: The value of `ADMIN_INIT_PASSWORD` in `.env` or `tier0` if not set. ## Next [Section titled “Next”](#next) * [Try the Demo Factory](../try-the-demo-factory-in-tier0/) - A preloaded plant to explore * [Choosing the Best Version](../choosing-the-best-product/) - Compare each version and select the one best fits you

# Choosing the Best Product

> Edge, Cloud, or Enterprise — positioning, plans, and a capability matrix for picking your Tier0 edition.

Check [here](https://tier0.app/pricing) for detailed features of each product and make the choice that fits you the best. ## Next [Section titled “Next”](#next) * [Build Apps on UNS](../../using-tier0/building-apps/) — Build and deploy Tier0 app on Enterprise * [Installation](../installation/)

# Try the Demo Factory in Tier0

> Explore a preloaded, working plant in your Cloud trial workspace — namespace, flows, notebooks, and apps.

A demo factory that already works on Tier0 shows you how the factory data is modeled, managed and used in a smart and smooth way. ## Business Background [Section titled “Business Background”](#business-background) The factory in Tier0 receives orders from ERP, splits them, schedules production, sends the schedule to production, and receives production progress data as work proceeds. 
```
%%{init: {"themeVariables": {"fontSize": "18px"}, "themeCSS": ".nodeLabel, .edgeLabel, .cluster-label { font-size: 18px !important; } .nodeLabel b, .nodeLabel strong { font-size: 19px !important; }"}}%%
flowchart TB
  uns["<span style='font-size:18px;font-weight:700'>UNS order data</span><br/><span style='font-size:16px'>Orders from ERP</span>"]
  production["<span style='font-size:18px'>Production Department</span>"]

  subgraph app["Work Order Management app"]
    direction TB
    appSpacer[" "]
    subgraph appPages[" "]
      direction LR
      orders["<span style='font-size:18px;font-weight:700'>Production Orders</span><br/><span style='font-size:16px'>(Optional) Create orders</span>"]
      plans["<span style='font-size:18px;font-weight:700'>Plans</span><br/><span style='font-size:16px'>Split order<br/>Production schedule</span>"]
      details["<span style='font-size:18px;font-weight:700'>Plan Details</span><br/><span style='font-size:16px;white-space:nowrap'>Production progress details</span>"]
    end
  end

  uns -- "order data" --> orders
  orders --> plans
  plans --> details
  plans --> planLabel["production plan"]
  planLabel --> production
  production --> progressLabel["production progress"]
  progressLabel --> details

  classDef external fill:#f8fafc,stroke:#b8c4d5,color:#171a22,stroke-width:1px,font-size:18px
  classDef optional fill:#fbfdf4,stroke:#cfe8a3,color:#171a22,stroke-width:1px,font-size:18px
  classDef appNode fill:#f4f8e9,stroke:#9adf11,color:#171a22,stroke-width:1px,font-size:18px
  classDef labelNode fill:transparent,stroke:transparent,color:#4b5563,font-size:17px
  classDef invisible fill:transparent,stroke:transparent,color:transparent
  class uns,production external
  class orders optional
  class plans,details appNode
  class planLabel,progressLabel labelNode
  class appSpacer invisible
  style appPages fill:transparent,stroke:transparent
```
 ## Process [Section titled “Process”](#process) 1. Go to **UNS** and check the details of orders that came from ERP. * `DemoFactory/ERP/ProductionOrders/State/UpsertProductionOrder`: Orders. * `DemoFactory/ERP/ProductionOrders/State/OrderList`: Current order list snapshot. 2. (optional) Go to **Launchpad**, access the **Work Order Management** application, and create orders on the **Production Orders** page. 3. In **Work Order Management**, split orders and schedule production plans for the split work orders on the **Plans** page. 4. Send the plan to production, and check the plan details on the following topics on **UNS**. * `DemoFactory/ERP/WorkOrderPlan/Metric/SplitCount`: The number of work orders after splitting. * `DemoFactory/ERP/WorkOrderPlan/State/PlanStatus`: Current plan status. * `DemoFactory/ERP/WorkOrderPlan/State/WorkOrderList`: The work order list after the production plan is scheduled. 5. Check the production progress data on the **Plan Details** page in the application. ## Next [Section titled “Next”](#next) * [Choosing the Best Version](../choosing-the-best-product/) - Compare each version and select the one best fits you * [UNS Concepts](../../using-tier0/factory-data-foundation/) - Understand data modeling in Unified Namespace

# Factory Data Foundation

> The Unified Namespace model: Metric, State, Action, and how the UNS stores data.

In Tier0, factory data is organized into clear, tree-structured models based on UNS (Unified Namespace) methodology, forming a unified data foundation. ## Data Foundation Architecture [Section titled “Data Foundation Architecture”](#data-foundation-architecture) * The broker receives and sends data through model topics. 
```
%%{init: {"flowchart": {"defaultRenderer": "elk", "nodeSpacing": 32, "rankSpacing": 48, "diagramPadding": 8}}}%%
flowchart TB

source["Data Source / Consumer"]

subgraph UNS["Unified Namespace"]
    direction LR

    Model["Model<br/>Defines paths,<br/>topics, and payloads"]
    Broker["Broker<br/>Handles data exchange"]
    Storage["Storage<br/>Persists data"]

    Model <--> Broker
    Broker --> Storage
end

source <-- "MQTT / API" --> UNS

classDef broker fill:#F7FAF2,stroke:#D8E6B8,stroke-width:1px,color:#2A2A2A,stroke-dasharray:5 5
class Broker broker
```
 * Models are categorized into three types, **Metric**, **State**, and **Action**. * Data is stored in different databases based on model type. * **Action/State**: JSONB (PostgreSQL) * **Metric**: Real-time data (TSDB) 
```
    %%{init: {
      "themeVariables": {
        "fontSize": "14pt"
      },
      "flowchart": {
        "defaultRenderer": "elk",
        "nodeSpacing": 40,
        "rankSpacing": 56,
        "diagramPadding": 8
      }
    }}%%
    flowchart LR
    subgraph Model["Model Type"]
        direction TB
        metric["Metric<br/>High-frequency numeric data"]
        state["State<br/>Current facts or status"]
        action["Action<br/>Commands or requested operations"]
    end
    subgraph Storage["Storage"]
        direction TB
        tsdb["TSDB<br/>Time-series storage<br/>for Metric history"]
        jsonb["JSONB<br/>Document storage<br/>for State and Action"]
    end
    metric --> tsdb
    state --> jsonb
    action --> jsonb
```
 ## Broker [Section titled “Broker”](#broker) The UNS broker is the platform component that receives and sends data through MQTT or OpenAPI based on data models. * **By MQTT**: The complete path of each node in the model tree works as the MQTT topic address to pass the message payload. * **By OpenAPI**: Perform CRUD commands on data models through HTTP requests. ## Model [Section titled “Model”](#model) ### Metric, State, Action [Section titled “Metric, State, Action”](#metric-state-action) Industrial data changes at different frequencies and serves different purposes, so Tier0 separates UNS models into three fixed types. | Type | Object | Example | | -------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------ | | `METRIC` | Designed for high-frequency data that is collected continuously and needs time-series storage. | Machine temperature/pressure/voltage | | `STATE` | Designed for data that changes less often and describes the current condition or fact of an object. | Machine operational status | | `ACTION` | Designed for lower-frequency command data that records what an external system or operator should execute. | Start batch, stop machine | ### Path/Topic, Payload [Section titled “Path/Topic, Payload”](#pathtopic-payload) Each UNS model contains path, topic, and payload. Together, they form a **tree** structure that carries business context and transmits industrial data. * **Path/Topic**: The path represents data ownership, and the topic carries the message payload. Together, they define the ***model address***. * **Payload**: The message payload held by the topic. It is JSON in a **key-value** format and defines the ***model data***. For example, a model in UNS looks like the tree displayed below: - Physical Data 
  ```plaintext
  Factory_A
  └── Site_01
      └── SMT_Line_1
          └── Metric
              └── Machine_001
  ```
 * **Path**: `Factory_A/Site_01/SMT_Line_1/Metric` * **Topic**: `/Machine_001`. * **Payload** 
    ```json
    "temperature": 85,
    "vibration": 2.8,
    ```
 - Business Data 
  ```plaintext
  Factory_A
  └── ERP
      └── ProductionOrders
          └── State
              └── OrderList
  ```
 * **Path**: `Factory_A/ERP/ProductionOrders/State` * **Topic**: `/OrderList`. * **Payload** 
    ```json
    "orders": "0123569876",
    "updated_at": 2026-7-13,
    ```
 ## Storage [Section titled “Storage”](#storage) Tier0 uses PostgreSQL to store data. Different data types change at different rates and have different storage requirements. | Data Type | Storage | Description | | --------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `METRIC` | TSDB | Optimized for high-frequency measurements such as temperature and pressure. Each topic includes default fields: `timestamp` and `quality`. | | `STATE` | PostgreSQL | Stores machine or system states in a flexible `JSONB` format. | | `ACTION` | PostgreSQL | Stores commands or actions in a flexible `JSONB` format. | ## Next [Section titled “Next”](#next) * [Connect Data to UNS](../preparing-data-foundation/) - How to model your plant to UNS and connect data to the model. * [Working with UNS Data](../working-with-factory-data/) - How to operate on your UNS model through MQTT and APIs.

# Preparing Data Foundation

> Model your namespace and bring data in with Source Flows. Available in all editions.

First build a model in **UNS**, then connect data through **Source Flow** to the model, forming a data foundation for subsequent use. 
```
flowchart BT
  uns["UNS Model"]
  flow["Source Flow"]


  flow -->|"send data"| uns

  classDef t0accent fill:#EFFFD2,stroke:#8AC926,stroke-width:1px,color:#18230B
  classDef t0node fill:#F8FAFC,stroke:#CBD5E1,stroke-width:1px,color:#111827
  class uns t0accent
  class flow t0node
```
 ## Build a Data Model [Section titled “Build a Data Model”](#build-a-data-model) Define the data hierarchy as a tree map according to: * The physical location the data comes from ([ISA-95](../../reference/isa-95-equipment-hierarchy/)); * The business logic the data conforms to. ### Build Models Manually [Section titled “Build Models Manually”](#build-models-manually) 1. Design the model structure, and add paths by order before adding the topic. 2. Set **Topic Type** among **Metric**, **State** and **Action** based on [data features](../factory-data-foundation/#metric-state-action). 3. Select **Mock Data** to send simulated data to the model, and **Enable History** to store the data to database. 4. Add data fields to the topic with the correct data types by **Auto Parsing** or **Pre-defined**. * **Auto Parsing**: Convert fields automatically from JSON text in batches. 
     ```json
     {
       "temp":85.5
     }
     ```
 Caution Can only parse single layer JSON, and for multi-layer JSON, only the first layer will be parsed. * **Pre-defined**: Manually set fields for the topic one by one. ### Import Models [Section titled “Import Models”](#import-models) 1. Send the template JSON from **Import** to AI, and use a similar prompt. 
   ```plaintext
   Generate a UNS model used for xx in xx plant, including xx equipment and data sources based on the template.
   ```
 2. Import the generated result in UNS. ## Connect Data to UNS [Section titled “Connect Data to UNS”](#connect-data-to-uns) ### Data Flow Features [Section titled “Data Flow Features”](#data-flow-features) * Based on Node-RED. * Every flow ends with an `mqtt out` node. It works as an MQTT client to publish data to the UNS broker. * The UNS broker is embedded in the `mqtt out` and `mqtt in` nodes with the same name as the flow. * When a UNS model is used as the topic, data goes directly to the corresponding model in **UNS**. ### Steps [Section titled “Steps”](#steps) 1. Use nodes based on the data source type to build a flow, and end it with an `mqtt out` node. 2. Make sure the **Server** of the `mqtt out` node is set to the UNS broker, and topic is a model from **UNS**. ### Flow Version Management [Section titled “Flow Version Management”](#flow-version-management) Save versions of the data flow, so you can either roll back to an earlier version or start a new flow with a history version. Caution Version rollback is not supported for Edge edition. ## Next [Section titled “Next”](#next) * [Build Apps on UNS](../building-apps/) — Build industrial applications with UNS data. * [Analyze UNS Data](../analyzing-data/) — Analyze UNS data with Marimo Notebook and Python.

# Manage Data from Center to Edge

> Seamlessly manage data on connected edge nodes from a central node.

In a real factory, data is often collected by edge devices and managed from a central node to reduce network bandwidth usage and improve data security. Tier0 provides a fleet management solution to manage data on edge nodes from a central node. ## Connecting Edge Nodes to the Center [Section titled “Connecting Edge Nodes to the Center”](#connecting-edge-nodes-to-the-center) 1. Create fleet nodes on the central node first. 2. Use the token generated by the central node to connect each edge node. ## Syncing Data between Edge and Center [Section titled “Syncing Data between Edge and Center”](#syncing-data-between-edge-and-center) * UNS data can be selected and synced only from edge nodes to the central node. Once synced, both history and real-time data of the selected topics will be available on the central node. * From the central node, you can assign users or user groups to edge nodes for node management. ## Next [Section titled “Next”](#next) * [Modeling Factory Data](../../best-practice/modeling-factory-data/) — Learn how to model factory data in UNS. * [Preparing Data Foundation](../preparing-data-foundation/) — Model your namespace and connect source data.

# Working with Factory Data

> Operate on the UNS data over MQTT and the API — with the API reference.

Data in Tier0 unified namespace can be altered and managed through MQTT and APIs with authorization. ## MQTT [Section titled “MQTT”](#mqtt) MQTT clients can connect to the UNS broker with credentials generated on Tier0. ### Generating MQTT Credentials [Section titled “Generating MQTT Credentials”](#generating-mqtt-credentials) On the **Edge** tab in Tier0, create a new credential and make a note of it. ### Connecting MQTT Clients [Section titled “Connecting MQTT Clients”](#connecting-mqtt-clients) * MQTTX 1. Install and open [MQTTX](https://mqttx.app/). 2. Add a connection with the information of the UNS broker and the generated credentials. 
     ```json
     {
       "Host": "mqtt.tier0.dev",
       "Port": 1883,
       "Client ID": "<generated client ID>",
       "Username": "<generated username>",
       "Password": "<generated password>"
     }
     ```
 * MQTT Explorer 1. Install and open [MQTT Explorer](https://mqtt-explorer.com/). 2. Add a new connection with the UNS broker information and the generated credentials. 
     ```json
     {
       "Protocol": "mqtt://",
       "Host": "mqtt.tier0.dev",
       "Port": 1883,
       "Username": "<generated username>",
       "Password": "<generated password>"
     }
     ```
 3. Change the client ID to the credential ID in **ADVANCED** before connecting. * Node-RED * Use **Event Flow** - Data processing module in Tier0 based on Node-RED. 1. Create an **Event Flow** in **Flows**, and start it with an `mqtt in` node. 2. Select the UNS broker as **Server** (same name as the flow) and set **Topic** to a UNS model in the node. * Use **Independent Node-RED Instance** 1. Install and open the [Node-RED](https://nodered.org/docs/getting-started/local) instance. 2. Add a server in the `mqtt in` node with the following information. 
    ```json
    {
      {/* under Connection */}
      "Server": "mqtt://mqtt.tier0.dev",
      "Port": 1883,
      "Client ID": "<generated clientId>",
      {/* under Security */}
      "Username": "<generated username>",
      "Password": "<generated password>"
    }
    ```
 ### Testing Connections [Section titled “Testing Connections”](#testing-connections) 1. Make sure the client status is normal. 2. Copy a UNS model topic and publish a message to it through the client. ## API [Section titled “API”](#api) Use the **REST API** to operate on UNS data. ### Getting API Key [Section titled “Getting API Key”](#getting-api-key) API keys are from **Settings > API Keys**. Depend on different account type, allowed key types are different. * **Service Key**: Can only be created by **Admin** and **Owner**. * **Personal Key**: Can be created by anyone, and the permission scope is the same as that of the account. Key permisson vs Account permission Account permission trumps key permission. When the key permission is set to **Full Access**, data unavailable to the account still cannot be accessed with this key. ### Examples [Section titled “Examples”](#examples) * `/uns/create` Create UNS path or topic. Use `namespace` to define the node tree. * JavaScript 
    ```js
    const BASE_URL = process.env.TIER0_BASE_URL;
    const API_KEY = process.env.TIER0_API_KEY;


    const response = await fetch(`${BASE_URL}/uns/create`, {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        'x-api-key': API_KEY,
      },
      body: JSON.stringify({
        namespace: [
          {
            name: 'DemoFactory',
            type: 'PATH',
            children: [
              {
                name: 'Site_01',
                type: 'PATH',
                children: [
                  {
                    name: 'Production',
                    type: 'PATH',
                    children: [
                      {
                        name: 'Produced_Qty',
                        type: 'TOPIC',
                        topicType: 'METRIC',
                        displayName: 'Produced Quantity',
                        description: 'Produced quantity from the production line.',
                        enableHistory: true,
                        fields: [
                          { name: 'value', type: 'DOUBLE', unit: 'pcs' },
                        ],
                      },
                    ],
                  },
                ],
              },
            ],
          },
        ],
      }),
    });


    const result = await response.json();
    console.log(result.data.results);
    ```
 * Python 
    ```python
    import os
    import requests


    base_url = os.environ["TIER0_BASE_URL"]
    api_key = os.environ["TIER0_API_KEY"]


    response = requests.post(
        f"{base_url}/uns/create",
        headers={"x-api-key": api_key},
        json={
            "namespace": [
                {
                    "name": "DemoFactory",
                    "type": "PATH",
                    "children": [
                        {
                            "name": "Site_01",
                            "type": "PATH",
                            "children": [
                                {
                                    "name": "Production",
                                    "type": "PATH",
                                    "children": [
                                        {
                                            "name": "Produced_Qty",
                                            "type": "TOPIC",
                                            "topicType": "METRIC",
                                            "displayName": "Produced Quantity",
                                            "description": "Produced quantity from the production line.",
                                            "enableHistory": True,
                                            "fields": [
                                                {"name": "value", "type": "DOUBLE", "unit": "pcs"},
                                            ],
                                        },
                                    ],
                                },
                            ],
                        },
                    ],
                },
            ],
        },
        timeout=30,
    )
    response.raise_for_status()


    result = response.json()
    print(result["data"]["results"])
    ```
 * cURL 
    ```bash
    curl -X POST "$TIER0_BASE_URL/uns/create" \
      -H "content-type: application/json" \
      -H "x-api-key: $TIER0_API_KEY" \
      -d '{
        "namespace": [
          {
            "name": "DemoFactory",
            "type": "PATH",
            "children": [
              {
                "name": "Site_01",
                "type": "PATH",
                "children": [
                  {
                    "name": "Production",
                    "type": "PATH",
                    "children": [
                      {
                        "name": "Produced_Qty",
                        "type": "TOPIC",
                        "topicType": "METRIC",
                        "displayName": "Produced Quantity",
                        "description": "Produced quantity from the production line.",
                        "enableHistory": true,
                        "fields": [
                          { "name": "value", "type": "DOUBLE", "unit": "pcs" }
                        ]
                      }
                    ]
                  }
                ]
              }
            ]
          }
        ]
      }'
    ```
 * `/uns/read` Read the latest value and metadata for one or more UNS topics. * JavaScript 
    ```js
    const BASE_URL = process.env.TIER0_BASE_URL;
    const API_KEY = process.env.TIER0_API_KEY;


    const response = await fetch(`${BASE_URL}/uns/read`, {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        'x-api-key': API_KEY,
      },
      body: JSON.stringify({
        topics: [
          'DemoFactory/Site_01/Production/Produced_Qty',
        ],
        include_leaf_value: true,
        include_metadata: true,
      }),
    });


    const result = await response.json();
    console.log(result.data.results);
    ```
 * Python 
    ```python
    import os
    import requests


    base_url = os.environ["TIER0_BASE_URL"]
    api_key = os.environ["TIER0_API_KEY"]


    response = requests.post(
        f"{base_url}/uns/read",
        headers={"x-api-key": api_key},
        json={
            "topics": [
                "DemoFactory/Site_01/Production/Produced_Qty",
            ],
            "include_leaf_value": True,
            "include_metadata": True,
        },
        timeout=30,
    )
    response.raise_for_status()


    result = response.json()
    print(result["data"]["results"])
    ```
 * cURL 
    ```bash
    curl -X POST "$TIER0_BASE_URL/uns/read" \
      -H "content-type: application/json" \
      -H "x-api-key: $TIER0_API_KEY" \
      -d '{
        "topics": [
          "DemoFactory/Site_01/Production/Produced_Qty"
        ],
        "include_leaf_value": true,
        "include_metadata": true
      }'
    ```
 * `/uns/update` Update a UNS node by `path`. Use `updateMask` to specify which fields should change. * JavaScript 
    ```js
    const BASE_URL = process.env.TIER0_BASE_URL;
    const API_KEY = process.env.TIER0_API_KEY;


    const response = await fetch(`${BASE_URL}/uns/update`, {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        'x-api-key': API_KEY,
      },
      body: JSON.stringify({
        path: 'DemoFactory/Site_01/Production/Produced_Qty',
        displayName: 'Produced Quantity',
        description: 'Updated production output quantity.',
        updateMask: ['displayName', 'description'],
      }),
    });


    const result = await response.json();
    console.log(result.data);
    ```
 * Python 
    ```python
    import os
    import requests


    base_url = os.environ["TIER0_BASE_URL"]
    api_key = os.environ["TIER0_API_KEY"]


    response = requests.post(
        f"{base_url}/uns/update",
        headers={"x-api-key": api_key},
        json={
            "path": "DemoFactory/Site_01/Production/Produced_Qty",
            "displayName": "Produced Quantity",
            "description": "Updated production output quantity.",
            "updateMask": ["displayName", "description"],
        },
        timeout=30,
    )
    response.raise_for_status()


    result = response.json()
    print(result["data"])
    ```
 * cURL 
    ```bash
    curl -X POST "$TIER0_BASE_URL/uns/update" \
      -H "content-type: application/json" \
      -H "x-api-key: $TIER0_API_KEY" \
      -d '{
        "path": "DemoFactory/Site_01/Production/Produced_Qty",
        "displayName": "Produced Quantity",
        "description": "Updated production output quantity.",
        "updateMask": ["displayName", "description"]
      }'
    ```
 * `/uns/delete` Delete one or more UNS models. Use `hard_delete: false` to move models to the recycle bin. * JavaScript 
    ```js
    const BASE_URL = process.env.TIER0_BASE_URL;
    const API_KEY = process.env.TIER0_API_KEY;


    const response = await fetch(`${BASE_URL}/uns/delete`, {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        'x-api-key': API_KEY,
      },
      body: JSON.stringify({
        topics: [
          'DemoFactory/Site_01/Production/Produced_Qty',
        ],
        hard_delete: false,
      }),
    });


    const result = await response.json();
    console.log(result.data);
    ```
 * Python 
    ```python
    import os
    import requests


    base_url = os.environ["TIER0_BASE_URL"]
    api_key = os.environ["TIER0_API_KEY"]


    response = requests.post(
        f"{base_url}/uns/delete",
        headers={"x-api-key": api_key},
        json={
            "topics": [
                "DemoFactory/Site_01/Production/Produced_Qty",
            ],
            "hard_delete": False,
        },
        timeout=30,
    )
    response.raise_for_status()


    result = response.json()
    print(result["data"])
    ```
 * cURL 
    ```bash
    curl -X POST "$TIER0_BASE_URL/uns/delete" \
      -H "content-type: application/json" \
      -H "x-api-key: $TIER0_API_KEY" \
      -d '{
        "topics": [
          "DemoFactory/Site_01/Production/Produced_Qty"
        ],
        "hard_delete": false
      }'
    ```
 ## API reference [Section titled “API reference”](#api-reference) * [Tier0 OpenAPI Reference](../../reference/tier0-openapi/) API key limitations on Tier0 Edge API key obtained from Tier0 Edge is only valid for UNS and Flows APIs. ## Next [Section titled “Next”](#next) * [Build Apps on UNS](../building-apps/) — Build industrial applications with UNS data. * [Analyze Data](../analyzing-data/) — Analyze UNS data with Marimo Notebook and Python.

# Building Apps

> A container is an app — deploy full containers on Tier0 Edge and Enterprise; build logic with Event Flow.

Build customized industrial applications through AI conversations. Applications can use factory data and include built-in capabilities such as role-based access control, version management, and external deployment. ## Building a Tier0 App [Section titled “Building a Tier0 App”](#building-a-tier0-app) Create a project before building an application. * Project is used to categorize applications and manage application access control. * Applications in one project can work as related functional modules of a complex system. * Data in each project is independent from other projects. ### Writing Prompt [Section titled “Writing Prompt”](#writing-prompt) Make sure your prompt covers ***at least*** the following aspects for the agent to generate an application that better fits your requirements. * **Application Purpose**: What is the purpose of the application? 
  ```text
  Create an equipment maintenance application for factory technicians.
  ```
 * **Application Data**: What data does the application use and where is it? * Platform Plan Applications can consume data from and publish data to UNS by stating the UNS topic in the prompt. 
    ```text
    The application should use equipment status, alarm records, and maintenance history from the xxx UNS model, and publish updates to the xxx UNS topic.
    ```
 * Builder Plan The application does not connect to UNS. Data generated during application usage is stored in the database. * **Application Function**: What does the application do? 
  ```text
  The application should allow users to monitor equipment conditions, submit maintenance requests, and track repair progress.
  ```
 When you are on **Builder** The **Builder** plan does not currently support UNS, but simulated data is available. For more specific data integration needs, please contact our team at <sales@freezonex.io>. ### Access Control [Section titled “Access Control”](#access-control) Role-based access control is applied to both projects and applications. * Project access levels * **Owner**: Full access to the project. * **Can Edit**: Full access to the project except deleting it. * **Can Launch**: Can only see the project in **Launchpad** and use allowed applications inside. * Application roles * Automatically generated by agent based on business logic of the application when not explicitly defined. Existing roles from other applications may be reused when their responsibilities align. * Can only be assigned to project members. * The application is open to all project members if the prompt explicitly specifies that no roles are required. Application role assignment is a **MUST** for role-controlled applications To access a role-controlled application in **Launchpad**, you must have application roles assigned. Even a workspace/project owner cannot access the application without an application role. ### Version Management [Section titled “Version Management”](#version-management) Applications exist on Tier0 in 3 shapes. * **Draft**: The latest state of the application. Changes on this version will not affect other versions if not deployed. * **Snapshot**: Saves the application at the time of taking the snapshot. Deploying a snapshot version takes down the current **Active** version. * **Active**: The current online version of the application. ### Export to External [Section titled “Export to External”](#export-to-external) Once the application is ready, you can export it from the **Preview** page for external third-party environments or Tier0 Enterprise. * **Download Source Code**: Export source code of the application for secondary development. * **Export Bundle**: Specifically exported for local import on **Tier0 Enterprise**. ## Next [Section titled “Next”](#next) * [Analyze Data](../analyzing-data/) — Analyze UNS data with Marimo Notebook and Python. * [Best practice: connecting industrial protocols](../../best-practice/connecting-industrial-protocols/) - Examples of common data source connections.

# Building Digital Twins

> Bind live UNS data to 3D models and assemble a complete factory scene for digital-twin visualization.

Factory data connected to Tier0 can be visualized as digital twin with life-like motions and environment. The process goes from mapping data to 3D models to combining models together to build a complete factory scene. ## Mapping Data to 3D Models [Section titled “Mapping Data to 3D Models”](#mapping-data-to-3d-models) 1. Select nodes to map with UNS data for real-time motions after uploading a model in **Model**. 2. Add instances with different UNS data onto the model to simulate different machines. 3. Bind data to the model, and use **Preview** to check whether it moves and rotates as the data changes. * **Position**: The model moves along the mapped direction of the coordinate. * **Rotation**: The model rotates around the coordinate original point in the mapped direction. 4. Use **Data Tagging** to add tags for static information display. ## Visualizing Data [Section titled “Visualizing Data”](#visualizing-data) 1. Add instances that should exist in the scene from **Instance Library**. 2. Use tools at the top to move, scale, and rotate models to fit the scene, and to change the viewing perspective and coordinate position. 3. Model position can be confirmed by **Object** at the right corner, which displays the bound UNS data. 4. Viewport and lighting are for environment details. ## External Share [Section titled “External Share”](#external-share) Only instances can be shared either by model or by scene. * **Share Model**: Generates a Tier0 platform link where the model with live UNS data is displayed. * **Share Scene**: Generates a Tier0 platform link where the model and its scene, with live UNS data and environment settings, are displayed.

# Analyzing Data

> Query live namespace data in Tier0 Notebook and build interactive analyses and machine-learning apps. Available in Cloud and Enterprise.

Marimo Notebook is used in Tier0 for advanced data analysis with **Python** scripts based on the factory data modeled and connected to UNS. 1. UNS data is under **VARIABLES** > **PostgreSQL** > **tenant\_id** > **uns**. 2. Use **@** to include data sources when you **GENERATE WITH AI**, and fuzzy match is available for data sources searching. 3. Save versions of your analysis for future rollbacks. ## Next [Section titled “Next”](#next) * [Build Apps on UNS](../building-apps/) — Build industrial applications with UNS data. * [UNS Concepts](../factory-data-foundation/) — Understand data modeling in Unified Namespace.

# Operating on Tier0 with Agents

> Use AI agents to operate Tier0 through Skills and CLI commands.

Tier0 provides Skills and CLI commands that allow AI agents to operate on the platform. **Work on Tier0 with your preferred AI agent.** Claude Code Codex Gemini CLI Cursor GitHub Copilot ## Installing Skills and CLI [Section titled “Installing Skills and CLI”](#installing-skills-and-cli) Choose the installer that matches your environment. ### With Node.js [Section titled “With Node.js”](#with-nodejs) If Node.js 16 or later is available, use `npx` to install the CLI and skills. 
```bash
npx -y @tier0/cli@latest install
```
 This runs the full installer and does two things: * Installs the `tier0` CLI binary into `~/.tier0/bin/`. * Installs the embedded Tier0 Skill baseline and synchronizes it to detected agents such as Codex, Claude Code, and more. ### Without Node.js [Section titled “Without Node.js”](#without-nodejs) 1. Install the `tier0` CLI binary. * curl 
     ```bash
     curl -fsSL https://raw.githubusercontent.com/FREEZONEX/Tier0-cli/main/install.sh | bash
     ```
 * iwr 
     ```powershell
     iwr https://raw.githubusercontent.com/FREEZONEX/Tier0-cli/main/install.ps1 | iex
     ```
 2. Download the latest Tier0 Skills and synchronize them to detected agents. 
   ```bash
   tier0 skills update
   ```
 3. Verify the installed Skill version and health. 
   ```bash
   tier0 skills status
   ```
 ### Managing Skills [Section titled “Managing Skills”](#managing-skills) * Check the installed Skill version and health: 
  ```bash
  tier0 skills status
  ```
 * Repair a missing or damaged embedded Skill baseline: 
  ```bash
  tier0 skills install
  ```
 * Update to the latest Tier0 Skill: 
  ```bash
  tier0 skills update
  ```
 ## How Skills and CLI Work Together [Section titled “How Skills and CLI Work Together”](#how-skills-and-cli-work-together) Tier0 CLI provides executable commands, and Skills guide agents in choosing and safely using these commands. ### Example [Section titled “Example”](#example) * Prompt: Check the current temperature of `Plant/Line1/Metric/Temperature`. * Skill * `tier0-uns` * It tells the agent that a current value should be read from a full topic path, and selects the appropriate UNS read operation. * CLI: `tier0 uns read Plant/Line1/Metric/Temperature --json` ### Skills [Section titled “Skills”](#skills) | Skill | Purpose | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `tier0` | Routes Tier0 tasks to the appropriate skill and provides shared setup, authentication, and safety rules. | | `tier0-uns` | Guides agents through UNS operations, including browse, read, write, history, search, create, update, delete, and restore. | | `tier0-flow` | Guides agents through SourceFlow and EventFlow management, including inspection, export, creation, update, deployment, and deletion. | | `tier0-mqtt` | Guides agents through MQTT credential management, publishing, and bounded real-time subscriptions. | | `tier0-files` | Guides agents through Tier0 file operations, including upload, download, access URL generation, and deletion. | ## CLI Operations [Section titled “CLI Operations”](#cli-operations) ### Setting Operating Instance [Section titled “Setting Operating Instance”](#setting-operating-instance) Private deployments must configure the base URL before login. Once it’s configured, all the following commands will be executed on that instance. 
```bash
tier0 config --base-url https://your-tier0.example.com
tier0 login # interactive browser login
```
 ### Authentication [Section titled “Authentication”](#authentication) * With an API key 1. Set the API key you got from the operating Tier0 instance. 
     ```bash
     tier0 config --api-key <api-key>
     ```
 2. Verify the API key identity and permissions. 
     ```bash
     tier0 auth whoami --json
     ```
 3. Perform other operations with the `tier0` CLI. * Without an API key 1. Get the browser authentication link. 
     ```bash
     tier0 login --no-wait --json
     ```
 2. Go through authentication on the browser by accessing the returned `verification_url`. 3. Poll for the login result with the returned `setup_code`. 
     ```bash
     tier0 login --setup-code <setup_code>
     ```
 Important Run `config` to set the operating instance before configuring an API key or starting browser login; otherwise the CLI may operate on the wrong instance. ### Previewing Operations [Section titled “Previewing Operations”](#previewing-operations) Use `--dry-run` to validate and preview supported write operations without sending a request to Tier0. #### Supported Operations [Section titled “Supported Operations”](#supported-operations) * UNS: `write/create/update/delete/restore` * Flow: `create/update/delete/deploy` * API: `tier0 api` * MQTT: `auth create/auth delete/publish` * Files: `delete` #### Examples [Section titled “Examples”](#examples) 
```bash
tier0 uns write --topic demo --value '{"value":1}' --dry-run
tier0 flow deploy --id 1 --flows-file flows.json --dry-run --json
tier0 api /openapi/v1/uns/write --body-file body.json --dry-run --json
tier0 mqtt auth create --name my-client --save my-client --dry-run --json
tier0 assets delete --file-path workspace/path/to/file.csv --dry-run --json
```
 Request body must contain valid JSON The CLI no longer guesses at or rewrites malformed JSON if no valid JSON is provided. #### Returned Previews [Section titled “Returned Previews”](#returned-previews) 
```json
{
  "ok": true,
  "dry_run": true,
  "data": {
    "api": [
      {"method": "POST", "url": "https://tier0.dev/openapi/v1/uns/write", "body": {}}
    ]
  }
}
```
 ### Handling Errors [Section titled “Handling Errors”](#handling-errors) With `--json`, command failures return a JSON object with the error message and code. 
```json
{"ok":false,"error":{"type":"validation","subtype":"invalid_argument","param":"--qos","message":"--qos must be 0, 1, or 2"}}
```
 ## Uninstalling Skills and CLI [Section titled “Uninstalling Skills and CLI”](#uninstalling-skills-and-cli) * Uninstallation Commands 
```bash
# Remove the Tier0 CLI while keeping local configuration and skills.
npx @tier0/cli@latest uninstall


# Remove the Tier0 CLI and delete local configuration, credentials and cache, but keep skills.
npx @tier0/cli@latest uninstall --purge


# Remove skills
npx @tier0/cli@latest uninstall --remove-skills


# Remove all of the above
npx @tier0/cli@latest uninstall --purge --remove-skills
```
 * Global NPM Uninstallation 
```bash
npm uninstall -g @tier0/cli
```
 * Manual Cleanup 
```bash
rm -rf ~/.tier0/bin/tier0 ~/.tier0/skills
npx -y --package=skills -- skills remove tier0 -y -g
```
 ## CLI Reference [Section titled “CLI Reference”](#cli-reference) [Tier0 CLI Reference](../../reference/cli-commands-reference/) lists all CLI commands. ## Next [Section titled “Next”](#next) * [Working with UNS Data](../working-with-factory-data/) — How to operate on your UNS model through MQTT and APIs.

# Modeling Factory Data

> The Demo UNS design: avoiding topic explosion and payload structure best practices.

Data in your factory, either machine data sending from sensors, or order data exiting in ERP/MES, can all be modeled into understandable tree structure in Tier0. ## How the Demo UNS is Structured [Section titled “How the Demo UNS is Structured”](#how-the-demo-uns-is-structured) ### Physical Data [Section titled “Physical Data”](#physical-data) Industrial data from machines and equipment with physical locations is modeled based on the [ISA-95 standard](../../reference/isa-95-equipment-hierarchy/). #### Demo Model [Section titled “Demo Model”](#demo-model) * ISA-95 Hierarchy 
  ```text
  Enterprise (Level 4)
  └── Site (Level 3/4)
      └── Area (Level 3)
          └── Work Center / Process Cell (Level 2/3)
              └── Equipment / Unit (Level 1/2)
                  └── Control Module / Device (Level 0/1)
  ```
 * Tree Structure 
  ```text
  DemoFactory
  └── Site_01
      └── Production
          └── Line_01
              └── WorkOrderExecution
                  ├── Metric
                  │   ├── Target_Qty
                  │   ├── Produced_Qty
                  │   ├── Defect_Qty
                  │   └── Completion_Rate
                  └── State
                      ├── CurrentWorkOrder
                      └── WorkOrderStatus
  ```
 * JSON 
  ```json
  {
    "namespace": [
      {
        "name": "DemoFactory",
        "alias": "DemoFactory_9e50bbcd833d",
        "type": "PATH",
        "children": [
          {
            "name": "Site_01",
            "alias": "DemoFactory__6d1fcad14e86",
            "type": "PATH",
            "children": [
              {
                "name": "Production",
                "alias": "__Site_01_Pro_91523d2ad304",
                "type": "PATH",
                "children": [
                  {
                    "name": "Line_01",
                    "alias": "__Production__acd5841c1ada",
                    "type": "PATH",
                    "children": [
                      {
                        "name": "WorkOrderExecution",
                        "alias": "__Line_01_Wor_54904ff56e91",
                        "type": "PATH",
                        "children": [
                          {
                            "name": "Metric",
                            "alias": "metric_233fd84305c4",
                            "type": "PATH",
                            "children": [
                              {
                                "name": "Target_Qty",
                                "alias": "Target_Qty_8646b092e543",
                                "type": "TOPIC",
                                "topicType": "METRIC",
                                "description": "Target quantity for the current Work Order.",
                                "fields": [
                                  {
                                    "name": "production_order_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "plan_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "plan_number",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "work_order_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "value",
                                    "type": "DOUBLE"
                                  },
                                  {
                                    "name": "unit",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "timestamp",
                                    "type": "DATETIME"
                                  }
                                ],
                                "enableHistory": "TRUE",
                                "mockData": "FALSE"
                              },
                              {
                                "name": "Produced_Qty",
                                "alias": "Produced_Qty_1f8cd3e11f77",
                                "type": "TOPIC",
                                "topicType": "METRIC",
                                "description": "Produced quantity for the current Work Order.",
                                "fields": [
                                  {
                                    "name": "production_order_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "plan_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "plan_number",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "work_order_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "value",
                                    "type": "DOUBLE"
                                  },
                                  {
                                    "name": "unit",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "timestamp",
                                    "type": "DATETIME"
                                  }
                                ],
                                "enableHistory": "TRUE",
                                "mockData": "FALSE"
                              },
                              {
                                "name": "Defect_Qty",
                                "alias": "Defect_Qty_b48fb3563ec1",
                                "type": "TOPIC",
                                "topicType": "METRIC",
                                "description": "Defect quantity for the current Work Order.",
                                "fields": [
                                  {
                                    "name": "production_order_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "plan_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "plan_number",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "work_order_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "value",
                                    "type": "DOUBLE"
                                  },
                                  {
                                    "name": "unit",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "timestamp",
                                    "type": "DATETIME"
                                  }
                                ],
                                "enableHistory": "TRUE",
                                "mockData": "FALSE"
                              },
                              {
                                "name": "Completion_Rate",
                                "alias": "Completion_R_7294e38e1530",
                                "type": "TOPIC",
                                "topicType": "METRIC",
                                "description": "Completion percentage for the current Work Order.",
                                "fields": [
                                  {
                                    "name": "production_order_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "plan_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "plan_number",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "work_order_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "value",
                                    "type": "DOUBLE"
                                  },
                                  {
                                    "name": "unit",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "timestamp",
                                    "type": "DATETIME"
                                  }
                                ],
                                "enableHistory": "TRUE",
                                "mockData": "FALSE"
                              }
                            ]
                          },
                          {
                            "name": "State",
                            "alias": "state_539a0f331fe0",
                            "type": "PATH",
                            "children": [
                              {
                                "name": "CurrentWorkOrder",
                                "alias": "CurrentWorkO_a638320784dd",
                                "type": "TOPIC",
                                "topicType": "STATE",
                                "description": "Current active Work Order on Line_01.",
                                "fields": [
                                  {
                                    "name": "production_order_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "plan_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "plan_number",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "work_order_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "target_line",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "workstation",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "product_code",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "target_qty",
                                    "type": "DOUBLE"
                                  },
                                  {
                                    "name": "unit",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "started_at",
                                    "type": "DATETIME"
                                  },
                                  {
                                    "name": "updated_at",
                                    "type": "DATETIME"
                                  }
                                ],
                                "enableHistory": "TRUE",
                                "mockData": "FALSE"
                              },
                              {
                                "name": "WorkOrderStatus",
                                "alias": "WorkOrderSta_e24e3bf113d1",
                                "type": "TOPIC",
                                "topicType": "STATE",
                                "description": "Execution status of the current Work Order.",
                                "fields": [
                                  {
                                    "name": "production_order_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "plan_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "plan_number",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "work_order_id",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "status",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "reason",
                                    "type": "STRING"
                                  },
                                  {
                                    "name": "updated_at",
                                    "type": "DATETIME"
                                  }
                                ],
                                "enableHistory": "TRUE",
                                "mockData": "FALSE"
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
  ```
 ### Business Data [Section titled “Business Data”](#business-data) For business planning and process objects, such as production orders and work order plans in an ERP, model data according to its business context. #### Business Data Modeling Guide [Section titled “Business Data Modeling Guide”](#business-data-modeling-guide) 
```plaintext
Factory / Business Domain / Business Object / Topic Type / Topic
```
 * **Business Domain**: The business process or system the data belongs to. 
  ```plaintext
  ERP
  MES
  WMS
  QMS
  EAM
  LIMS
  Planning
  Inventory
  Maintenance
  Quality
  Logistics
  ...
  ```
 * **Business Object**: The core managed object in the business domain. 
  ```plaintext
  ProductionOrders
  WorkOrderPlan
  MaterialInventory
  QualityInspection
  MaintenanceOrders
  PurchaseOrders
  Shipments
  ...
  ```
 * **Topic Type**: **Metric**, **Action**, and **State**. | Topic Type | Used for | Examples | | ---------- | ------------------------------------ | ------------------------------------------------------------------------- | | STATE | Current status or current facts | Work order status, order status, equipment status | | METRIC | Continuously changing numeric values | Temperature, speed, output, OEE, energy consumption | | ACTION | Commands for external executors | Start equipment, stop equipment, reset alarm, request work order dispatch | #### Demo Model [Section titled “Demo Model”](#demo-model-1) * Tree Structure 
  ```text
  DemoFactory
  └── ERP
      ├── ProductionOrders
      │   └── State
      │       ├── UpsertProductionOrder
      │       └── OrderList
      └── WorkOrderPlan
          ├── Metric
          │   └── SplitCount
          └── State
              ├── PlanStatus
              └── WorkOrderList
  ```
 * JSON 
  ```json
  {
    "namespace": [
      {
        "name": "DemoFactory",
        "alias": "DemoFactory_f925c1ec5041",
        "type": "PATH",
        "children": [
          {
            "name": "ERP",
            "alias": "DemoFactory__e504805b1973",
            "type": "PATH",
            "children": [
              {
                "name": "ProductionOrders",
                "alias": "__ERP_Product_1e7984ac3a1d",
                "type": "PATH",
                "children": [
                  {
                    "name": "State",
                    "alias": "state_c2616c8698fa",
                    "type": "PATH",
                    "children": [
                      {
                        "name": "UpsertProductionOrder",
                        "alias": "UpsertProduc_d5f166da35c6",
                        "type": "TOPIC",
                        "topicType": "STATE",
                        "description": "Event for creating or updating a Production Order from WOM or ERP simulation.",
                        "fields": [
                          {
                            "name": "event_id",
                            "type": "STRING"
                          },
                          {
                            "name": "event_type",
                            "type": "STRING"
                          },
                          {
                            "name": "source",
                            "type": "STRING"
                          },
                          {
                            "name": "production_order_id",
                            "type": "STRING"
                          },
                          {
                            "name": "order_number",
                            "type": "STRING"
                          },
                          {
                            "name": "product_code",
                            "type": "STRING"
                          },
                          {
                            "name": "planned_qty",
                            "type": "DOUBLE"
                          },
                          {
                            "name": "unit",
                            "type": "STRING"
                          },
                          {
                            "name": "status",
                            "type": "STRING"
                          },
                          {
                            "name": "due_date",
                            "type": "DATETIME"
                          },
                          {
                            "name": "updated_at",
                            "type": "DATETIME"
                          }
                        ],
                        "enableHistory": "TRUE",
                        "mockData": "FALSE"
                      },
                      {
                        "name": "OrderList",
                        "alias": "OrderList_11fbca51a2bd",
                        "type": "TOPIC",
                        "topicType": "STATE",
                        "description": "Retained compatibility snapshot of current Production Orders.",
                        "fields": [
                          {
                            "name": "orders",
                            "type": "STRING"
                          },
                          {
                            "name": "updated_at",
                            "type": "DATETIME"
                          }
                        ],
                        "enableHistory": "TRUE",
                        "mockData": "FALSE"
                      }
                    ]
                  }
                ]
              },
              {
                "name": "WorkOrderPlan",
                "alias": "__ERP_WorkOrd_19f04960daf5",
                "type": "PATH",
                "children": [
                  {
                    "name": "Metric",
                    "alias": "metric_17258b809a4f",
                    "type": "PATH",
                    "children": [
                      {
                        "name": "SplitCount",
                        "alias": "SplitCount_032e86f5fbfc",
                        "type": "TOPIC",
                        "topicType": "METRIC",
                        "description": "Number of executable Work Orders generated from a Work Order Plan.",
                        "fields": [
                          {
                            "name": "production_order_id",
                            "type": "STRING"
                          },
                          {
                            "name": "plan_id",
                            "type": "STRING"
                          },
                          {
                            "name": "plan_number",
                            "type": "STRING"
                          },
                          {
                            "name": "split_count",
                            "type": "INTEGER"
                          },
                          {
                            "name": "updated_at",
                            "type": "DATETIME"
                          }
                        ],
                        "enableHistory": "TRUE",
                        "mockData": "FALSE"
                      }
                    ]
                  },
                  {
                    "name": "State",
                    "alias": "state_884cee548bf7",
                    "type": "PATH",
                    "children": [
                      {
                        "name": "PlanStatus",
                        "alias": "PlanStatus_2e21c1327c49",
                        "type": "TOPIC",
                        "topicType": "STATE",
                        "description": "Current lifecycle status of a Work Order Plan.",
                        "fields": [
                          {
                            "name": "production_order_id",
                            "type": "STRING"
                          },
                          {
                            "name": "plan_id",
                            "type": "STRING"
                          },
                          {
                            "name": "plan_number",
                            "type": "STRING"
                          },
                          {
                            "name": "status",
                            "type": "STRING"
                          },
                          {
                            "name": "updated_at",
                            "type": "DATETIME"
                          }
                        ],
                        "enableHistory": "TRUE",
                        "mockData": "FALSE"
                      },
                      {
                        "name": "WorkOrderList",
                        "alias": "WorkOrderLis_887b92f26fc2",
                        "type": "TOPIC",
                        "topicType": "STATE",
                        "description": "Split result list generated from a sent Work Order Plan.",
                        "fields": [
                          {
                            "name": "production_order_id",
                            "type": "STRING"
                          },
                          {
                            "name": "plan_id",
                            "type": "STRING"
                          },
                          {
                            "name": "plan_number",
                            "type": "STRING"
                          },
                          {
                            "name": "split_count",
                            "type": "INTEGER"
                          },
                          {
                            "name": "lines",
                            "type": "STRING"
                          },
                          {
                            "name": "updated_at",
                            "type": "DATETIME"
                          }
                        ],
                        "enableHistory": "TRUE",
                        "mockData": "FALSE"
                      }
                    ]
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
  ```
 ## Payload Structure [Section titled “Payload Structure”](#payload-structure) Data to be consumed and written back must both be considered when structuring the topic payload. * **Action** Action payloads automatically include `_timestamp`, which records when the data is stored in the Tier0 database. * start\_job 
    ```json
    {
      "light_turn_on": true,
      "height_adjust": 12.5,
      "_timestamp": "2026-07-10T10:00:00Z"
    }
    ```
 * stop\_job 
    ```json
    {
      "water_pump_off": true,
      "window_close": true,
      "_timestamp": "2026-07-10T10:00:00Z"
    }
    ```
 * **Metric** Metric payloads automatically include `_timestamp`, which records when the data is stored in the Tier0 database, and `_quality`, which represents data quality. * ERP / SplitCount 
    ```json
    {
      "production_order_id": "PO-20260710-001",
      "plan_id": "PLAN-001",
      "plan_number": "WOP-20260710-001",
      "split_count": 3,
      "updated_at": "2026-07-10T10:00:00Z",
      "_timestamp": "2026-07-10T10:00:00Z",
      "_quality": "Good"
    }
    ```
 * Site\_01 / Target\_Qty 
    ```json
    {
      "production_order_id": "PO-20260710-001",
      "plan_id": "PLAN-001",
      "plan_number": "WOP-20260710-001",
      "work_order_id": "WO-001",
      "value": 1200,
      "unit": "pcs",
      "timestamp": "2026-07-10T10:00:00Z",
      "_timestamp": "2026-07-10T10:00:00Z",
      "_quality": "Good"
    }
    ```
 * **State** State payloads automatically include `_timestamp`, which records when the data is stored in the Tier0 database. * ERP / UpsertProductionOrder 
    ```json
    {
      "event_id": "EVT-001",
      "event_type": "UPSERT",
      "source": "ERP",
      "production_order_id": "PO-20260710-001",
      "order_number": "PO-001",
      "product_code": "SKU-1001",
      "planned_qty": 1200,
      "unit": "pcs",
      "status": "Released",
      "due_date": "2026-07-12T00:00:00Z",
      "updated_at": "2026-07-10T10:00:00Z",
      "_timestamp": "2026-07-10T10:00:00Z"
    }
    ```
 * Site\_01 / WorkOrderStatus 
    ```json
    {
      "production_order_id": "PO-20260710-001",
      "plan_id": "PLAN-001",
      "plan_number": "WOP-20260710-001",
      "work_order_id": "WO-001",
      "status": "Running",
      "reason": "In production",
      "updated_at": "2026-07-10T10:00:00Z",
      "_timestamp": "2026-07-10T10:00:00Z"
    }
    ```
 ## How to Avoid Topic Explosion [Section titled “How to Avoid Topic Explosion”](#how-to-avoid-topic-explosion) Tier0 uses three fixed topic types, `Metric`, `Action`, and `State`, to avoid topic explosion. * **Use `Metric` for high-frequency real-time data** Put high-frequency measurements and telemetry under `Metric` instead of expanding the namespace for every signal detail. * Bad 
    ```text
    Factory_A
    └── SMT_Line_1
        └── Machine_001
            ├── Temperature_Value
            ├── Temperature_Unit
            ├── Temperature_Status
            └── Speed_Value
    ```
 * Better 
    ```text
    Factory_A
    └── SMT_Line_1
        └── Metric
            └── Machine_001
    ```
 Topic payload: 
    ```json
    {
      "temperature": 80,
      "temperature_unit": "Celsius",
      "temperature_status": "Normal",
      "speed": 300
    }
    ```
 * **Use `State` for low-frequency status data** Put low-frequency status, snapshots, and current business state under `State`. Avoid creating a new topic for every changing entity such as jobs, batches, or orders. * Bad 
    ```text
    Factory_A
    └── SMT_Line_1
        └── State
            ├── Job_001
            ├── Job_002
            └── Job_003
    ```
 * Better 
    ```text
    Factory_A
    └── SMT_Line_1
        └── State
            └── Machine_001
    ```
 Topic payload: 
    ```json
    {
      "current_job": "Job_003",
      "product": "Product_A"
    }
    ```
 * **Use `Action` for low-frequency action data** Put commands, requests, and operator actions under `Action`. Keep action inputs as fields on a topic rather than creating one topic per parameter. * Bad 
    ```text
    Factory_A
    └── SMT_Line_1
        └── Machine_001
            ├── Start_Command
            ├── Target_Height
            └── Light_On
    ```
 * Better 
    ```text
    Factory_A
    └── SMT_Line_1
        └── Action
            └── StartJob
    ```
 Topic payload: 
    ```json
    {
      "target_height": 12.5,
      "light_on": true,
      "requested_by": "operator"
    }
    ```

# Connecting Industrial Protocols

> Connect OPC UA, Modbus, and other typical protocol devices into the UNS with Node-RED — with real flow JSON references.

This section uses common data source types, including OPC UA, Modbus, and APIs, as examples to demonstrate how to connect data sources to **UNS** through **Source Flow**. ## OPC UA [Section titled “OPC UA”](#opc-ua) ### Configuring Data Flow [Section titled “Configuring Data Flow”](#configuring-data-flow) 1. In a new **Source Flow**, add `inject`, `OPC UA Client` and `mqtt out` nodes. 2. Connect them in order and configure their details. * `inject` Add a property entry and set the properties to the following items: 
  ```json
  {
    "msg.payload": "[
                      {
                          "name": "D103_CHLORIDE",
                          "nodeId": "ns=2;i=3",
                          "datatype": "Float"
                      },
                      {
                          "name": "D103_PH",
                          "nodeId": "ns=2;i=4",
                          "datatype": "Float"
                      },
                      ...
                  ]",
    "msg.topic": "multiple"
  }
  ```
 * `OPC UA Client` 
  ```json
  "Endpoint": "<OPC UA server address and port>", //e.g. opc.tcp://127.0.0.1:4850
  "Action": "SUBSCRIBE",
  "Interval": "<data collecting interval>" //e.g. 10 seconds
  ```
 * `mqtt out` 
  ```json
  "Server": "<embedded UNS broker>", //same name as the flow
  "Topic": "<UNS model>",
  ```
 ### Flow Reference [Section titled “Flow Reference”](#flow-reference) ## Modbus [Section titled “Modbus”](#modbus) ### Configuring Data Flow [Section titled “Configuring Data Flow”](#configuring-data-flow-1) In the flow, add `modbus-read` and `mqtt out` nodes and configure the required information. * `modbus-read` * **Settings** * **FC**: Function code that specifies the Modbus action, e.g., FC 3 = Read Holding Registers. * **Address**: The starting register address to read from (usually zero-based). * **Quantity**: The number of consecutive registers to read. * **Poll Rate**: How often the node polls the Modbus device (e.g., every 10 seconds). * **Server**: Reference to a configured Modbus server (IP, port, protocol, etc.). * **Server** * **Host**: Modbus server IP. * **Port**: Modbus server port. * `mqtt out` 
  ```json
  "Server": "<embedded UNS broker>", //same name as the flow
  "Topic": "<UNS model>",
  ```
 ### Flow Reference [Section titled “Flow Reference”](#flow-reference-1) ## API [Section titled “API”](#api) ### Configuring Data Flow [Section titled “Configuring Data Flow”](#configuring-data-flow-2) In the flow, add `inject`, `http request` and `mqtt out` nodes and configure the required information. * `http request` 
  ```json
  "Method": "<API method>",
  "URL": "<API URL>",
  "Return": "a parsed JSON object",
  "Headers": "<API headers>"
  ```
 * `mqtt out` 
  ```json
  "Server": "<embedded UNS broker>", //same name as the flow
  "Topic": "<UNS model>",
  ```
 ### Flow Reference [Section titled “Flow Reference”](#flow-reference-2)

# Building Shopfloor Workflow with Agent

> Use the UNS agent to build workflows that cover data modeling, data connection, and data events.

UNS agent provides a faster and easier way to build factory data models, connect data from factories and create alarm events based on the data through conversation in natural language. ## Workflow Background [Section titled “Workflow Background”](#workflow-background) A mixing tank mixes materials while heating them. Build a data model and collect temperature, water level, and heater status to determine whether the tank is overheated and trigger an alarm. 
```
flowchart RL
  source["Source Flow<br/>(connect data sources)"]
  event["Event Flow<br/>(event logic and alarm)"]
  uns["UNS<br/>(modeling)"]

  source -->|"mixing tank data"| uns
  event -->|"alarm data"| uns

  classDef t0accent fill:#EFFFD2,stroke:#8AC926,stroke-width:1px,color:#18230B
  classDef t0node fill:#F8FAFC,stroke:#CBD5E1,stroke-width:1px,color:#111827
  class uns t0accent
  class source,event t0node
```
 ## How to Build the Workflow [Section titled “How to Build the Workflow”](#how-to-build-the-workflow) 1. Start a conversation with the UNS agent in **UNS** with `full_access`. 2. Enter the prompt to build UNS models. 
   ```text
   Create a data model representing the temperature, water level and heater status of a mixing tank, temperature and level in one metric topic and heater status in a state topic.
   ```
 3. Once you confirm that the model is complete, enter the prompt and let the agent connect the corresponding data to the model. 
   ```text
   Create a source flow to send data to these 2 topics, simulate reasonable data.
   ```
 4. Enter the prompt to create an event with the following logic. * Low water level warning: Triggered when the water level is below 20% and the heater is on. * Dry-heating alarm: Triggered when the water level is below 15%, temperature is above 90°C and the heater is on. 
   ```md
   Create an Event Flow for the following alarm logic:
     - Warning: Trigger when level < 20 and heater_status = true.
     - Critical: Trigger when level < 15, temperature > 90, and heater_status = true.
     - Clear the active alarm when level > 25 or heater_status = false.
   Create a new topic under the existing UNS model to receive the alarm result. Include the alarm level, message, active status, and timestamp in the output.
   ```
 5. Check the results on **UNS**.

# CLI Commands Reference

> CLI and agent skill command reference.

Search commands, flags, or examples... Global flagsThese flags can be used with any `tier0` command. `--json`Output raw JSON`--debug`Print HTTP request/response details ## Basic operations ## Skills ## Auth ## UNS ## Node-RED Flow ## Assets No matching CLI commands.

# Embedded Builder Skills

> Reference for the AI skills embedded in Tier0 Builder.

Tier0 App Builder currently includes 7 active embedded AI skills. These skills help the agent turn requirements into industrial applications, align applications with the Tier0 platform scaffold, repair UI issues, and keep preview environments stable. ## `app-srs` [Section titled “app-srs”](#app-srs) **Meaning**: The requirements-analysis entry point for new industrial-enterprise apps. It is used for WMS, MES, ERP, CRM, QMS, EAM, approval, inventory, supplier, customer, scheduling, and similar enterprise workflows. **Example**: 
```text
Use $app-srs to design a quality exception management system for an automotive parts factory.
```
 Natural language can also trigger it: 
```text
Generate a warehouse management app with receiving, putaway, picking, shipping, and inventory counting.
```
 **What it does**: 1. Analyzes business roles, workflows, states, and exceptions. 2. Checks whether the requirement is clear enough. 3. Generates a complete SRS. 4. Saves it to `specs/spec.md` by default. 5. Hands the result to `app-builder` by default unless the user asks for requirements only. ## `app-builder` [Section titled “app-builder”](#app-builder) **Meaning**: The core skill for actually building the app. **Example**: 
```text
Use $app-builder to build the first version from specs/spec.md.
```
 **Internal work types**: The agent decides which work type to use based on the prompt and current project state. * **First Release**: Builds the first version from an SRS. * **Delta Feature**: Adds a substantial module or workflow to an existing app. * **Light Edit**: Handles small field, copy, style, or validation changes without running the full build flow. **What it checks**: * Visible entry points. * Executable user paths. * Data or state changes. * Role permissions. * Build and preview verification. ## `align-platform-app` [Section titled “align-platform-app”](#align-platform-app) **Meaning**: A migration skill for aligning an existing app with the Tier0 platform scaffold. 
```text
Use $align-platform-app to migrate this old Vite React project into the Tier0 platform scaffold.
```
 **Suitable sources**: * Vite, CRA, Next.js, React Router, static HTML, and full-stack Node/React apps. * Older Tier0 apps. * Apps that rely on LocalStorage, IndexedDB, Zustand persist, in-memory mocks, or simulated data. **What it does**: * Aligns the app to the Tier0 TanStack Start scaffold. * Migrates persistent business data to Drizzle/PostgreSQL. * Aligns routes, APIs, services, and database setup. * Removes app-owned login, logout, and role-switching flows. * Connects to Gateway-owned identity and roles. * Preserves platform contracts such as `/api/health`, `/api/manifest`, and `artifact.toml`. * Verifies preview and production startup. ## `migrate-node-red-app` [Section titled “migrate-node-red-app”](#migrate-node-red-app) **Meaning**: A Node-RED migration skill for aligning a Node-RED application with the Tier0 platform scaffold. It is usually used together with `align-platform-app`. **Example**: 
```text
Use $align-platform-app and $migrate-node-red-app to migrate this Node-RED backend into a Tier0 TanStack Start app.
```
 **What it does**: * Inventories flows, subflows, function nodes, change nodes, and switch nodes. * Analyzes HTTP, MQTT, database, scheduled jobs, and external API calls. * Builds a migration mapping table. * Separates CRUD, workflow, scheduling, integration, and temporary simulation behavior. * Moves business logic into services. * Moves interfaces into API routes. * Moves state into PostgreSQL. * Moves long-running work and integrations into workers, queues, or external services. * Verifies browser, CRUD, workflow, and integration cutover separately. Caution It does not handle ordinary preview or packaging issues by itself. ## `ui-repair` [Section titled “ui-repair”](#ui-repair) **Meaning**: A repair skill for existing apps. It is used for UI quality issues, not for first-time app generation. **Example**: 
```text
Use $ui-repair to fix overflow and form misalignment in the order edit dialog on small screens.
```
 **Suitable issues**: * Misaligned forms or required markers. * Dialog or drawer overflow. * Accidental horizontal scrolling. * Clipped select menus or dropdowns. * Broken table or chart layout. * Flat pages with weak visual hierarchy. * Mixed UI languages. * PDA, kiosk, dashboard, and industrial interface layout problems. Caution The skill should locate the smallest owning component and fix the affected surface without rebuilding the whole app. For complex responsive work, use `responsive-craft`. ## `responsive-craft` [Section titled “responsive-craft”](#responsive-craft) **Meaning**: A responsive-layout skill for audits, new responsive builds, and multi-breakpoint previews. **Invocation modes**: * **audit**: Audits and repairs an existing responsive layout. * **build**: Builds a responsive page from scratch. * **preview**: Starts a multi-breakpoint live preview. **Examples**: 
```text
Use $responsive-craft audit to fix the mobile layout of this dashboard.
```
 
```text
Use $responsive-craft build to create an inventory workspace for phone, tablet, and desktop.
```
 
```text
Use $responsive-craft preview to show 375, 768, 1024, and 1440 px breakpoints.
```
 **Execution styles**: * **Adaptive**: Moves quickly and confirms design forks as they appear. * **Guided**: Writes responsive behavior specs for each component before implementation. ## `preview-runtime-stability` [Section titled “preview-runtime-stability”](#preview-runtime-stability) **Meaning**: A runtime troubleshooting skill for preview failures during app development. It does not build business features. **Example**: 
```text
Use $preview-runtime-stability to diagnose "Preview loading timed out".
```
 **Suitable issues**: * Missing or partial `node_modules`. * npm `ERESOLVE`. * Vite or TanStack Start startup failures. * Port `5173` conflicts. * `/api/health` failures. * SDK SSR module errors. * `artifact.toml` preview port mismatch. * Frequent full-page reloads during HMR. * Stale Node or Vite processes that did not exit. **Standard flow**: 1. Check dependency installation. 2. Check port `5173`. 3. Probe the health endpoint. 4. Reuse the preview when it is healthy. 5. Clean up the old preview when it is unhealthy. 6. Restart and verify the port, health endpoint, and HMR behavior. ## Skill Selection Guide [Section titled “Skill Selection Guide”](#skill-selection-guide) * Use `app-srs` first when the requirement is unclear or when you need a durable specification. * Use `app-builder` when the implementation scope is already clear. * Use `align-platform-app` and `migrate-node-red-app` for migration work. * Use `ui-repair`, `responsive-craft`, or `preview-runtime-stability` when the app already exists and the problem is visual, responsive, or runtime-specific.

# ISA-95 Equipment Hierarchy

> Reference levels for modeling physical assets and control systems with ISA-95.

The ISA-95 Equipment Hierarchy outlines how physical assets and control systems are structured within an organization. It is broken into the following levels: * **Enterprise**: The top level, representing the entire organization. This includes overall business processes and decision-making entities, typically supported by ERP systems. * **Site**: Represents a specific geographic location or facility. Each site can have its own management systems and local production control. * **Area**: An operational division within a site, often a functional area such as packaging, assembly, or warehousing. * **Process Cell**: The basic unit of manufacturing within an area. It encompasses all the equipment required to produce a specific product or perform a specific function. * **Unit**: A distinct part of the process cell, responsible for a specific function such as mixing, heating, or assembling. * **Equipment Module**: A modular piece of equipment that performs a specific task within a unit. For example, in a mixing unit, the module could be a heating element. * **Control Module**: The lowest level, representing individual devices or controllers such as sensors, pumps, actuators, or PID control loops that handle direct control of physical processes.

# Standard Port List

> Firewall port list for the standard six-VM offline Tier0 Enterprise deployment.

The following table is based on the Enterprise New six-VM offline deployment. Customer firewalls should allow traffic from the minimum required source ranges only. Do not expose these ports directly to the public internet. | Source | Target | Port | Protocol | Purpose | | ---------------------------------------- | --------------------------------- | ----------- | -------- | ------------------------------------------------------------------- | | Control host | Six target VMs | 22 | TCP | SSH / Ansible | | Three application nodes, mutually | Three application nodes | 2377 | TCP | Docker Swarm management plane | | Three application nodes, mutually | Three application nodes | 7946 | TCP/UDP | Swarm node discovery | | Three application nodes, mutually | Three application nodes | 4789 | UDP | Swarm overlay | | Three application nodes | Active application node | 8088 | TCP | HAProxy to Enterprise New backend; application network segment only | | Business network segment | APP VIP | 80 | TCP | Platform HTTP entry, `/healthz`, `/readyz` | | Business network segment | APP VIP / active application node | 1883 | TCP | MQTT | | Business network segment | APP VIP / active application node | 8883 | TCP | MQTT TLS | | Business network segment | APP VIP / active application node | 8083 | TCP | MQTT WebSocket | | Business network segment | APP VIP / active application node | 8084 | TCP | MQTT WebSocket TLS | | Business network segment | APP VIP / active application node | 4840 | TCP | OPC UA Server | | Controlled application network segment | Active application node | 8089 | TCP | Launchpad | | Dynamic app access network segment | Active application node | 30000-30199 | TCP | App Gateway / dynamic application host ports | | Application nodes, DB nodes, and witness | DB VIP / DB nodes | 5432 | TCP | PostgreSQL / TimescaleDB | | DB1, DB2, and witness | DB1, DB2, and witness | 5432 | TCP | Streaming replication / repmgr | | Application nodes | NFS server | 2049 | TCP/UDP | Used when `NFS_ENABLED=true` | The current compose deployment does not expose Redis, SourceFlow, EventFlow, Marimo, or the EMQX Dashboard externally. These services are only called by the backend inside the Docker runtime network.

# Operations Runbook

> Daily inspection, failover, backup, restore, cleanup, and troubleshooting procedures for the standard Tier0 Enterprise deployment.

This document is based on the Enterprise New offline application. ## Daily Inspection [Section titled “Daily Inspection”](#daily-inspection) 1. Run the following commands from the deployment package root directory: 
   ```bash
   ./bin/verify.sh --site config/site.conf
   ./bin/smoke-business-emqx.sh --site config/site.conf
   ```
 2. Confirm the following items: * The 3 Swarm nodes are `Ready`. * `APP VIP /healthz` and `/readyz` return `200`. * `DB VIP:5432` is accessible. * The PostgreSQL primary, standby, and witness topology is normal. * `backend`, `redis`, `emqx`, `sourceflow`, `eventflow`, `marimo`, and `opcua-server` are all running on the active application node. ## Application Checks [Section titled “Application Checks”](#application-checks) 1. Run the following commands on the active application node: 
   ```bash
   cd /opt/tier0-enterprise/tier0-deploy
   bash bin/compose.sh ps
   bash bin/compose.sh logs --tail=200 backend
   curl -fsS http://127.0.0.1:8088/healthz
   curl -fsS http://127.0.0.1:8088/readyz
   ```
 2. Health-check interpretation: * `healthz` failure: check the backend process, image, and container logs first. * `readyz` failure: also check DB VIP, Redis, and database migration logs. * Home page reachable but login unavailable: confirm `ADMIN_INITIAL_PASSWORD` in `.env`, then check backend IAM migration logs and user tables. Do not judge deployment success only from the HTTP port. ## Protocol Entry Checks [Section titled “Protocol Entry Checks”](#protocol-entry-checks) 
```bash
nc -vz <APP_VIP> 1883
nc -vz <APP_VIP> 8883
nc -vz <APP_VIP> 8083
nc -vz <APP_VIP> 8084
nc -vz <APP_VIP> 4840
```
 ## Database Checks [Section titled “Database Checks”](#database-checks) Run the following on database nodes: 
```bash
systemctl status tier0-postgresql
systemctl status tier0-repmgrd
sudo -iu postgres psql -Atqc 'select pg_is_in_recovery();'
sudo -iu postgres /usr/local/bin/repmgr -f /etc/repmgr.conf cluster show
```
 * `pg_is_in_recovery=false` means primary. * `pg_is_in_recovery=true` means standby. * DB VIP must only be located on the current primary. * If standby replication is interrupted or the witness reports upstream errors, preserve the site first. Do not rerun the full deployment directly. ## APP VIP Failover [Section titled “APP VIP Failover”](#app-vip-failover) 1. Before testing, record the node that currently owns the VIP and verify: 
   ```bash
   curl -fsS http://<APP_VIP>/healthz
   curl -fsS http://<APP_VIP>/readyz
   ```
 2. After stopping `keepalived` on the current VIP node, confirm VIP takeover on another application node, then retest both URLs. The HAProxy backend points to the active application node on `8088/TCP`, so all three application nodes must be able to reach port 8088 on the active node. 3. After restoring `keepalived`, run `verify.sh` and smoke checks again. Do not stop active application containers during the drill. ## DB Primary Failure Switchover [Section titled “DB Primary Failure Switchover”](#db-primary-failure-switchover) Before a failure drill, create a backup and confirm replication is healthy: 
```bash
./bin/backup-db.sh --site config/site.conf
```
 After stopping `tier0-postgresql` on the current primary, observe `repmgr` promoting the standby and DB VIP takeover. The target RTO is within 2 minutes. Historical tests on the same architecture were about 59-64 seconds. Caution * The old primary must not be started directly back into the production topology. Stop PostgreSQL, repmgrd, and keepalived first; remove any remaining DB VIP; then run basebackup from the current new primary, register the node as a standby, and finally reconnect the witness. * After failover, do not rerun the full installer directly. Static primary/standby definitions in the inventory may disturb the current topology. ## Backup and Restore [Section titled “Backup and Restore”](#backup-and-restore) * Backup: 
  ```bash
  ./bin/backup-db.sh --site config/site.conf
  ```
 * Restore: 
  ```bash
  ./bin/restore-db.sh --site config/site.conf --restore-file /backup/tier0-db/postgres-YYYYmmddTHHMMSS.dump
  ```
 ## Cleanup and Retest [Section titled “Cleanup and Retest”](#cleanup-and-retest) 
```bash
./full-cleanup.sh --env install.env --yes
./install.sh --auto --yes
```
 Caution Full cleanup deletes Tier0 runtime data, Docker/Swarm data, and database backups on the 6 nodes. It also performs final cleanup on the control node after the control container exits. Use this entry only for test environments that are confirmed disposable. If customer data or an incident site must be preserved, use normal `--cleanup` and complete a backup first. ## Common Issues [Section titled “Common Issues”](#common-issues) ### Backend Startup Failure [Section titled “Backend Startup Failure”](#backend-startup-failure) 
```bash
cd /opt/tier0-enterprise/tier0-deploy
bash bin/compose.sh logs --tail=300 backend
bash bin/compose.sh ps
```
 ### EMQX Startup Failure [Section titled “EMQX Startup Failure”](#emqx-startup-failure) 
```bash
cd /opt/tier0-enterprise/tier0-deploy
bash bin/compose.sh logs --tail=200 emqx
```
 ### repmgrd Repeated Restarts [Section titled “repmgrd Repeated Restarts”](#repmgrd-repeated-restarts) 
```bash
systemctl status tier0-repmgrd
tail -100 /var/log/repmgr/repmgr.log
sudo -iu postgres psql -Atqc 'show shared_preload_libraries;'
```
 ### Material Verification Failure [Section titled “Material Verification Failure”](#material-verification-failure) 
```bash
./bin/verify-materials.sh
```
 Caution Do not skip SHA256 verification and continue deployment. Re-acquire the delivery package or rebuild the materials following the traceable process.

# SLA and High Availability Boundaries

> SLA recommendations, high availability boundaries, acceptance gates, drills, and hardening items for the standard Tier0 Enterprise deployment.

This document applies to the six-VM fully offline deployment: three application / Swarm nodes, two PostgreSQL HA nodes, and one witness node. ## Architecture Baseline [Section titled “Architecture Baseline”](#architecture-baseline) | Layer | Current Design | High Availability Boundary | | ------------------- | --------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | Application runtime | Enterprise New compose runs on the fixed `tier0_app_active_node`. | `backend`, `redis`, `EMQX`, `SourceFlow`, `EventFlow`, `Marimo`, and `OPC UA Server` are not automatically multi-replica services yet. | | HTTP entry | Three application nodes run keepalived APP VIP + HAProxy. | The VIP can float. HAProxy forwards to the active node on `8088/TCP` and checks `/healthz`. | | MQTT / OPC UA | EMQX uses `1883/8883/8083/8084`; OPC UA uses `4840`. | These connect directly to the active node. They are not included in TCP VIP proxying, so protocol-entry HA is not promised. | | Database | PostgreSQL / TimescaleDB streaming replication + repmgr + witness + DB VIP. | If the primary fails, the standby can be promoted automatically. Asynchronous replication cannot promise strict zero RPO. | | Backup and restore | `backup-db.sh` / `restore-db.sh`. | Provides recoverability, but it is not the same as online HA. | ## Recommended SLA [Section titled “Recommended SLA”](#recommended-sla) | Service Object | Recommended Statement | Constraints | | ------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | Platform HTTP entry | Start from monthly availability of `99.5%`. | VIP failover is not the same as automatic migration of the active application node. | | Database access | Operations target: `RTO <= 2 minutes`. | RPO depends on replication lag. Historical failover tests on the same architecture were about 59-64 seconds. | | MQTT / OPC UA | No HA SLA is promised in the current design. | HAProxy TCP proxying, a separate VIP, or service clustering must be added first. | | Backup and restore | Daily backups; mandatory backup before major changes. | Restore time depends on data volume and on-site I/O. | Before claiming `99.9%` or higher, complete application automatic migration or multi-replica deployment, protocol-entry HA, monitoring alerts, and recovery drills. Do not describe planned capabilities as delivered capabilities. ## Acceptance Gates [Section titled “Acceptance Gates”](#acceptance-gates) Each deployment or upgrade must satisfy at least the following: * `verify-materials.sh` passes, and the delivery package SHA256 checks pass. * The three Swarm nodes are `Ready`. * `APP VIP /healthz` and `/readyz` return `200`, and the home page and `/uns` are accessible. * The seven compose services on the active node are running. * `DB VIP:5432` is accessible, and the primary, standby, and witness topology is normal. * The default account can actually log in. Do not judge IAM initialization success only from ports, containers, or HTTP 200 responses. * Smoke checks for HTTP, EMQX ports, WebSocket/WSS, and OPC UA TCP pass. ## Inspection and Alerts [Section titled “Inspection and Alerts”](#inspection-and-alerts) Run the following every day: 
```bash
./bin/verify.sh --site config/site.conf
./bin/smoke-business-emqx.sh --site config/site.conf
```
 Configure at least the following alerts: * APP VIP `/healthz` or `/readyz` failure. * DB VIP unreachable, no primary, split-brain risk, replication interruption, or replication lag above threshold. * keepalived, HAProxy, PostgreSQL, or repmgrd abnormal status. * Key containers such as `backend`, `redis`, and `EMQX` exit or restart frequently. * Data disk usage exceeds `80% / 90%`. * The latest backup failed or was not generated. ## Drill Requirements [Section titled “Drill Requirements”](#drill-requirements) | Drill | Frequency | Verification Criteria | | --------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------------- | | APP VIP failover | Before delivery and quarterly. | After stopping keepalived on the current VIP node, `/healthz` and `/readyz` recover. | | DB primary failure | Before delivery and every six months. | The standby is promoted, DB VIP recovers, and the old primary is rebuilt as a standby. | | Database restore | Monthly sample and before major changes. | The backup is restorable. After restore, `verify`, smoke checks, and login verification pass. | | Material verification | Every package build. | Source materials, package SHA, and unpacking verification pass. | ## Incident Handling Principles [Section titled “Incident Handling Principles”](#incident-handling-principles) 1. Confirm the impact scope first: HTTP, MQTT, OPC UA, database, and single-node impact. 2. Prioritize restoring VIP, HAProxy, the active application, and the database primary. 3. Preserve logs, data directories, and backups. Do not directly clean or overwrite the incident site. 4. For database incidents, determine the current primary first, then decide whether to fail over, rejoin, or restore. 5. After recovery, run `verify`, smoke checks, and actual login verification. ## High-SLA Hardening Items [Section titled “High-SLA Hardening Items”](#high-sla-hardening-items) * Make the application multi-replica, or establish a verifiable automatic migration process for the active node. * Cluster EMQX, or route it through HAProxy TCP / TLS with a dedicated VIP. * Provide a high-availability entry for OPC UA through HAProxy TCP, a dedicated VIP, or service clustering. * Use PostgreSQL synchronous replication, or define acceptable replication lag and RPO clearly. * Automate backup scheduling, off-site replication, and regular restore drills. * Connect containers, hosts, databases, logs, and black-box probes to unified alerting. Related procedures: [Operations Runbook](../operations-runbook/) and [Standard Port List](../standard-port-list/).

# API Reference

> Static OpenAPI endpoint summary

[OpenAPI Reference](../tier0-openapi/)

# Tier0 Docs

> Start building on the Tier0 Unified Namespace — paste one prompt into your code agent, then follow the best practices.

Start building with Tier0 ## Paste this into your code agent One prompt installs the Tier0 CLI + skill, logs you in, and takes your agent on a first lap around the Unified Namespace. 
```
Run `npx -y @tier0/cli@latest install` to install the tier0 CLI and agent skill.
Read https://docs.tier0.app/llms.txt for the full Tier0 docs.
```
 Copy prompt Prefer doing it by hand? [Installation guide](/get-started/installation/)[Demo Factory](/get-started/try-the-demo-factory-in-tier0/) Machine-readable endpoints for agents: [/llms.txt](/llms.txt) ## Best Practice Field-tested patterns from real Tier0 deployments. [Modeling](/best-practice/modeling-factory-data/) ### [Modeling Factory Data](/best-practice/modeling-factory-data/) [How the Demo UNS is designed — hierarchy, avoiding topic explosion, and payload structure that scales.](/best-practice/modeling-factory-data/) [Read more](/best-practice/modeling-factory-data/)[Protocols](/best-practice/connecting-industrial-protocols/) ### [Connecting Industrial Protocols](/best-practice/connecting-industrial-protocols/) [Node-RED flows for typical industrial protocols, with real importable flow JSON references.](/best-practice/connecting-industrial-protocols/) [Read more](/best-practice/connecting-industrial-protocols/)[Security](/best-practice/security-and-resilience/) ### [Security and Resilience](/best-practice/security-and-resilience/) [Protect identities and data access, choose encrypted transports, and understand deployment and recovery boundaries.](/best-practice/security-and-resilience/) [Read more](/best-practice/security-and-resilience/)[Analytics](/best-practice/building-analytics-apps/) ### [Building Analytics Apps](/best-practice/building-analytics-apps/) [From Notebook query to interactive analysis app — the Aramco Bowtie case, on sanitized data.](/best-practice/building-analytics-apps/) [Read more](/best-practice/building-analytics-apps/)[Digital Twin](/best-practice/displaying-data-with-digital-twin/) ### [Displaying Data with Digital Twin](/best-practice/displaying-data-with-digital-twin/) [Use Digital Twin scenes to visualize live UNS data in a workshop-style operating view.](/best-practice/displaying-data-with-digital-twin/) [Read more](/best-practice/displaying-data-with-digital-twin/)

# Security and Resilience

> Overview of identity, data access, deployment availability, and recovery controls in Tier0.

Tier0 ensures data security through the following aspects: * **Security**: Controls who can access the platform, applications, and data. * **Resilience**: Covers service continuity and data recovery after failures. ## Security Scope by Edition [Section titled “Security Scope by Edition”](#security-scope-by-edition) | Edition | Security Coverage | | ---------------------- | ------------------------------------------------------------------------------------------------------------- | | **Cloud** | Platform access control and authenticated access to MQTT and APIs. | | **Enterprise** | Platform and data access control, plus deployment availability, backup, and recovery for private deployments. | | **Edge (open source)** | Local data connection and modeling. Its built-in MQTT broker allows unauthenticated subscriptions. | See [Choosing the Best Product](https://tier0.app/pricing) for edition positioning. ## Security [Section titled “Security”](#security) ### Identity and Access Control [Section titled “Identity and Access Control”](#identity-and-access-control) | Area | Tier0 Mechanism | Details | | ------------------------------ | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | Platform access | * Cloud controls access through workspace membership. * Enterprise controls access through user accounts. | [User Management](../../reference/user-management/) | | Agent access | Agents authenticate with an API key and use the Tier0 CLI to access and operate the platform. | [CLI authentication](/using-tier0/operating-on-tier0-with-agents/#authentication) | | Project and application access | Tier0 uses RBAC to control access to applications. | [Access control](/using-tier0/building-apps/#access-control) | ### Data Access Control [Section titled “Data Access Control”](#data-access-control) | Access | Tier0 Mechanism | Details | | -------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | MQTT | * Clients authenticate with credentials generated by Tier0. * TLS and secure WebSocket encrypt data in transit. | [MQTT credentials and connections](/using-tier0/working-with-factory-data/#generating-mqtt-credentials) | | REST API | Accessing data requires a key generated by Tier0. | [API Reference](../../reference/api-reference/) | ## Resilience [Section titled “Resilience”](#resilience) ### Deployment Availability [Section titled “Deployment Availability”](#deployment-availability) Enterprise currently provides two deployment plans for center nodes: | Plan | Availability Boundary | Details | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------- | | **Standard deployment** | * Provides database primary/standby failover. * The Enterprise application and EMQX remain single points of failure. | [Standard Deployment Plan](../../reference/standard-deployment-plan/) | | **High availability deployment** | - Adds active/standby Enterprise nodes and defined failover behavior for the database, cache, object storage, and managed runtime files. - Does not promise zero interruption or zero data loss. | [High Availability Deployment Plan](../../reference/high-availability-deployment-plan/) | ### Backup and Recovery [Section titled “Backup and Recovery”](#backup-and-recovery) Enterprise provides database backup and recovery. After recovery, check deployment status, run basic functionality checks, and confirm that users can sign in to the platform. * For backups, see [Backup and Restore](/reference/operations-runbook/#backup-and-restore). * For recovery verification requirements, see [Drill Requirements](/reference/sla-and-ha-boundaries/#drill-requirements).

# Reference

> API, skill, deployment, and other technical references for Tier0.

Use this section for technical references that customers may want to open directly. * [API Reference](./api-reference/) * [CLI Command Reference](./cli-commands-reference/) * [Embedded Builder Skills](./embedded-builder-skills/) * [User Management](./user-management/) * [Standard Deployment Plan](./standard-deployment-plan/) * [High Availability Deployment Plan](./high-availability-deployment-plan/) * [SLA and High Availability Boundaries](./sla-and-ha-boundaries/) * [Standard Port List](./standard-port-list/) * [Operations Runbook](./operations-runbook/)

# High Availability Deployment Plan

> Reference deployment plan for a highly available Tier0 Enterprise environment.

## Plan Objective [Section titled “Plan Objective”](#plan-objective) This V2 plan deploys a logical Tier0 Enterprise Fleet Center in an offline or private environment. Six machines share the business, middleware, database, storage, and agent roles. Users and internal Enterprise services always connect through fixed VIP addresses. After one machine or one proxy fails, Keepalived, HAProxy, repmgr, RustFS, and Fleet Agent take over at their own layers. The plan targets single-machine failure recovery in one network segment. It is an active-standby disaster recovery design, not a multi-data-center, consensus-election, or zero-data-loss high availability cluster. The design goals are: * The Fleet Center entry can move to a healthy Standby node after the current business node fails. * PostgreSQL, Redis, and RustFS use fixed VIP addresses, so Enterprise does not need to change `.env` when backend roles change. * Local business files for App, Flow, Marimo, License, and similar runtime assets can be synchronized from Active to Standby. * One Cluster License maps to one logical installation and is not activated again after Enterprise switches nodes. * A separate deployer initializes the six machines, assigns roles, deploys services, monitors status, and validates failover. * VIP failover, file synchronization, independent Redis instances, database replication, and object-storage redundancy are clearly separated. This plan applies only to Fleet Center. Branch Enterprise nodes are still installed as normal single-machine deployments and do not join this six-machine active-standby design. ## Overall Topology [Section titled “Overall Topology”](#overall-topology) ![Tier0 Enterprise overall topology](/images/enterprise-ha-overview.en.svg) The core request path is: 
```text
Clients / site systems
    -> Business VIP :8088
    -> Current Active Enterprise (D / E / F)
    |-- PostgreSQL VIP :5432 -> D/E HAProxy -> A/B current writable Primary
    |-- Redis VIP      :6379 -> C or F independent Redis
    `-- RustFS VIP     :19000 -> A/B HAProxy -> C/D/E/F RustFS cluster
```
 The four VIPs are fixed entries in the business configuration. When Enterprise switches nodes or middleware backend roles change, clients and Enterprise do not change connection addresses. ## Planned IP Addresses [Section titled “Planned IP Addresses”](#planned-ip-addresses) The following addresses are examples only. Replace them with real addresses from the same on-site IPv4 subnet before implementation. The four VIPs must not be occupied by DHCP, static hosts, or other VRRP instances. | Name | Example IP | Description | | ------------------- | -------------- | --------------------------------------------------------------------- | | deployer-controller | `10.60.10.20` | Independent deployer; not counted as one of the six business machines | | ent-ha-01 / A | `10.60.10.11` | PostgreSQL Primary and related roles | | ent-ha-02 / B | `10.60.10.12` | PostgreSQL Standby and related roles | | ent-ha-03 / C | `10.60.10.13` | Witness, Redis-1, RustFS-1 | | ent-ha-04 / D | `10.60.10.14` | Initial Active Enterprise and related roles | | ent-ha-05 / E | `10.60.10.15` | Enterprise Standby and related roles | | ent-ha-06 / F | `10.60.10.16` | Enterprise Standby and related roles | | Business VIP | `10.60.10.101` | Unified Enterprise entry, default `8088` | | PostgreSQL VIP | `10.60.10.102` | Unified database entry, default `5432` | | Redis VIP | `10.60.10.103` | Unified Redis entry, default `6379` | | RustFS VIP | `10.60.10.104` | Unified S3 entry, default `19000` | Before implementation, confirm the business NIC name, IPv4 prefix, gateway, DNS, NTP, hostnames, SSH user, disk mount points, and firewall allowlist. VIPs, node IPs, and NICs must not be guessed by scripts and used directly in production. ## Six-Machine Responsibilities [Section titled “Six-Machine Responsibilities”](#six-machine-responsibilities) ![Tier0 Enterprise six-machine active-standby architecture](/images/enterprise-ha-six-node.en.svg) | Machine | Default deployed components | Key responsibility | | ------- | ---------------------------------------------------------------- | ---------------------------------------------------------- | | A | PostgreSQL Primary, RustFS HAProxy, Keepalived | Initial database primary; first RustFS proxy candidate | | B | PostgreSQL Standby, RustFS HAProxy, Keepalived | Streaming database standby; second RustFS proxy candidate | | C | PostgreSQL Witness, Redis-1, RustFS-1 | Database witness/status record; Redis and RustFS data node | | D | PG HAProxy, Keepalived, RustFS-2, Enterprise, Fleet Agent, rsync | Initial business Active; first PostgreSQL proxy candidate | | E | PG HAProxy, Keepalived, RustFS-3, Enterprise, Fleet Agent, rsync | Business Standby; second PostgreSQL proxy candidate | | F | Redis-2, RustFS-4, Enterprise, Fleet Agent, rsync | Business Standby; second Redis instance | This allocation is for single-machine failure tolerance and balanced resource usage across the six machines. The deployer maps machines to A-F by the natural order of hostnames/IP addresses and selects default roles, but the final assignment must still be reviewed against the on-site disk layout and failure domains. ### Machine Configuration Checklist [Section titled “Machine Configuration Checklist”](#machine-configuration-checklist) The Enterprise business layer is single-active/standby. At any time, only the node holding the Business VIP serves business requests. Scale the business layer vertically by increasing CPU, memory, and disk on one business node. Do not estimate concurrency by multiplying the number of Enterprise nodes. | Machine | Node type | vCPU | Memory | Base storage | Disk recommendation | | ------- | ------------------------------ | ---- | ------ | ------------ | -------------------------------------------------------------------------------------------- | | A | PostgreSQL Primary node | 8C | 16G | 1T | `/` about 80G; remaining space for PostgreSQL data, WAL, and backup staging | | B | PostgreSQL Standby node | 8C | 16G | 1T | `/` about 80G; remaining space for PostgreSQL data, WAL, and backup staging | | C | Witness / middleware node | 4C | 8G | 500G | Allocate fixed mount points for system, Redis, and RustFS data | | D | Initial Active Enterprise node | 8C | 32G | 500G | Allocate fixed mount points for system, containers, Enterprise synced files, and RustFS data | | E | Standby Enterprise node | 8C | 32G | 500G | Same as D | | F | Standby Enterprise node | 8C | 32G | 500G | Same as D | | Category | Count | Per-node spec | vCPU subtotal | Memory subtotal | Storage subtotal | | ------------------------- | ----- | ----------------- | ------------- | --------------- | ---------------- | | PostgreSQL database nodes | 2 | `8C / 16G / 1T` | 16C | 32G | 2T | | Witness / middleware node | 1 | `4C / 8G / 500G` | 4C | 8G | 500G | | Enterprise business nodes | 3 | `8C / 32G / 500G` | 24C | 96G | 1.5T | | **Total** | **6** | - | **44C** | **136G** | **4T** | ## Configuration Summary [Section titled “Configuration Summary”](#configuration-summary) ### Unified Enterprise Connection Configuration [Section titled “Unified Enterprise Connection Configuration”](#unified-enterprise-connection-configuration) D, E, and F keep the same Enterprise configuration. | Dependency | Configuration principle | | -------------- | -------------------------------------------------------------------------------- | | PostgreSQL | Connect to PostgreSQL VIP, not A/B node IPs | | Redis | Connect to Redis VIP, not C/F node IPs | | File storage | Use `FILESTORE_DRIVER=s3`; point the S3 endpoint to RustFS VIP | | Fleet identity | Three nodes share one logical `installationId`; each node has its own `memberId` | | Fleet Agent | One process-level Fleet Agent runs on each Enterprise node | | License | Activate once through the Business VIP; include the Bundle in the HA fileset | `installationId` and `memberId` are generated and persisted by the installation process. Operators should not invent them manually or copy `memberId` between machines. ### VIP and Candidate Nodes [Section titled “VIP and Candidate Nodes”](#vip-and-candidate-nodes) | VIP | Candidate nodes | Health basis | Takeover result | | -------------- | --------------- | --------------------------------------------------------------------- | --------------------------------------------------------------------- | | Business VIP | D, E, F | Fleet Agent readiness, business container state, sync generation | Only the Owner starts the Enterprise HA business group | | PostgreSQL VIP | D, E | HAProxy/Keepalived state; HAProxy routes only to the writable Primary | Enterprise keeps the same database address | | Redis VIP | C, F | Redis `PING` and local service state | Switches to the other independent instance; cache/session may be lost | | RustFS VIP | A, B | HAProxy and RustFS backend health | Proxy entry moves while the S3 address stays unchanged | ## Component Deployment Design [Section titled “Component Deployment Design”](#component-deployment-design) ### Enterprise, Fleet Agent, and Business VIP [Section titled “Enterprise, Fleet Agent, and Business VIP”](#enterprise-fleet-agent-and-business-vip) * D, E, and F install the same full Enterprise offline ZIP. * Fleet Agent is a host-level process service, not a business container. * Keepalived manages the Business VIP and calls the Fleet Agent readiness endpoint to decide whether the local node can hold the VIP. * Only the Business VIP Owner runs the business group; the other two nodes stay Standby. * A node that loses the VIP must stop the business group. A recovered node must rejoin as Standby first. * EMQX runs as part of the single-active business group. Do not restart it outside Agent control or build a separate election mechanism across the three machines. ### PostgreSQL Primary/Standby, Witness, and Unified Entry [Section titled “PostgreSQL Primary/Standby, Witness, and Unified Entry”](#postgresql-primarystandby-witness-and-unified-entry) * A starts as Primary, B starts as Streaming Standby, and C runs repmgr Witness. * Witness records cluster state and participates in failure decisions. It does not store full business data and does not serve reads or writes. * When the primary fails, repmgr promotes a healthy Standby to the new Primary. * D/E HAProxy routes `5432` traffic only to the current writable Primary. * PostgreSQL VIP floats between the D/E proxy nodes. Proxy failover and database primary/standby failover are decoupled. * Acceptance must prove both “only one writable Primary” and “Standby resumes streaming”. ### Redis Independent Instances [Section titled “Redis Independent Instances”](#redis-independent-instances) * C and F each run one standalone Redis instance. * The two Redis instances do not form Sentinel, Cluster, or primary/replica replication. * Redis VIP only switches the connection entry. When the current instance fails, the VIP moves to the other healthy instance. * Redis may only store data that can be lost or rebuilt. Empty cache and user re-login after switching are within the design boundary. * If Redis later stores non-discardable state, upgrade to a replication/arbitration design instead of keeping the independent-instance assumption. ### RustFS Four-Node Cluster [Section titled “RustFS Four-Node Cluster”](#rustfs-four-node-cluster) * C, D, E, and F form a four-node distributed RustFS object storage cluster. * HAProxy on A and B proxies all healthy RustFS backends. * RustFS VIP floats between A and B. Enterprise always accesses S3 through this VIP. * A single RustFS node failure is handled by the cluster mechanism; a single proxy failure is handled by VIP failover. * Do not synchronize object data with rsync, hot copy, or direct edits to RustFS mount directories. * RustFS node count, disk count, and erasure-coding availability boundaries must follow the cluster check results of the deployed version. ### App, Flow, Marimo, and License File Synchronization [Section titled “App, Flow, Marimo, and License File Synchronization”](#app-flow-marimo-and-license-file-synchronization) Fleet Agent synchronizes only local business files that cannot be placed directly in PostgreSQL/RustFS but must exist on the new Active node: * App runtime directories and `node_modules` * Flow, Marimo, Notebook, and related local runtime files * License Bundle * Other HA filesets explicitly registered by the current implementation Synchronization uses system `rsync` over deployer-managed SSH trust between business nodes. Each release creates a complete generation/manifest. A Standby node can take over only after confirming that the generation is complete. PostgreSQL data directories, Redis data directories, and RustFS data directories are not part of this synchronization scope. ## Connection Methods and Network Baseline [Section titled “Connection Methods and Network Baseline”](#connection-methods-and-network-baseline) ### Access Relationships [Section titled “Access Relationships”](#access-relationships) | Source | Target | Purpose | | ---------------------- | ------------------------ | --------------------------------------------------------------- | | Users / site systems | Business VIP | Web/API business access | | D/E/F Enterprise | PostgreSQL VIP | Business database reads/writes | | D/E/F Enterprise | Redis VIP | Cache and session | | D/E/F Enterprise | RustFS VIP | S3 object reads/writes | | Deployer controller | A-F | SSH, deployment, status collection, operations | | D/E/F | D/E/F | rsync/SSH file synchronization | | A/B/C | A/B/C | PostgreSQL streaming replication, repmgr, Witness communication | | RustFS members/proxies | C/D/E/F | RustFS cluster and S3 backend communication | | Nodes sharing a VIP | Corresponding candidates | VRRP advertisement and VIP movement | ### Ports and Protocols [Section titled “Ports and Protocols”](#ports-and-protocols) Allow ports by minimum required source-to-target relationships. Do not expose all management ports directly to the user network. | Port / protocol | Suggested allowlist | Purpose | | --------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------- | | `22/TCP` | Deployer -> A-F; D/E/F mutual sync | SSH, deployment, rsync | | `8088/TCP` | User network -> Business VIP | Enterprise business entry | | `5432/TCP` | D/E/F -> PostgreSQL VIP; PG/proxy internal access | PostgreSQL, streaming replication, proxy | | `6379/TCP` | D/E/F -> Redis VIP; health-check nodes -> C/F | Redis access and health checks | | `19000/TCP` | D/E/F -> RustFS VIP | Enterprise fixed S3 entry | | `9000/TCP` | A/B -> C/D/E/F; RustFS members | RustFS S3 backend/cluster communication; confirm by version | | `9001/TCP` | Controlled operations network -> RustFS admin entry | Optional RustFS admin endpoint; do not expose to user network | | `19731/TCP` | Localhost / Keepalived / controlled operations network | Fleet Agent HTTP health and control interface | | `18080/TCP` | Operations network -> deployer controller | Deployer HTTPS UI | | `112/VRRP` | Candidate nodes for each VIP | Keepalived VIP movement; this is an IP protocol number, not TCP/UDP | | `1883/8883` | Site device network -> business entry, per product configuration | MQTT/MQTTS | ## Failover and Capability Boundaries [Section titled “Failover and Capability Boundaries”](#failover-and-capability-boundaries) | Failure scenario | Expected behavior | Boundary | | ----------------------------- | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | | D Active business node fails | Business VIP moves to E or F; the new Owner starts the business group after readiness passes | Requires the latest HA fileset generation to be complete on Standby | | PostgreSQL A Primary fails | B is promoted to Primary; PostgreSQL VIP continues routing to the writable Primary through D/E proxy | Data loss depends on streaming replication state | | Redis current node fails | Redis VIP moves to the other independent Redis instance | Cache/session may be lost | | RustFS proxy A or B fails | RustFS VIP moves to the other proxy | RustFS data still depends on the four-node cluster health | | Single RustFS data node fails | RustFS cluster continues service if its redundancy conditions are met | Actual boundary depends on disk count and erasure-coding configuration | Network partition, simultaneous multi-node failure, incorrect VRRP priority, full disks, clock drift, and manual changes outside the deployer are outside the normal automatic recovery promise and must be covered by operations procedures. ## Backup, Monitoring, and Inspection [Section titled “Backup, Monitoring, and Inspection”](#backup-monitoring-and-inspection) * PostgreSQL requires independent logical or physical backup and restore verification. * RustFS requires capacity, disk, and cluster health monitoring. * Redis must be treated as cache/session unless a future replication design is introduced. * Fleet Agent and Keepalived state must be monitored on D/E/F. * rsync generation lag must be monitored. A Standby with stale generation must not take over business traffic. * Regular failover drills must include business VIP, PostgreSQL promotion, Redis VIP switch, RustFS proxy switch, and restore from backup. ## Pre-Go-Live Checks [Section titled “Pre-Go-Live Checks”](#pre-go-live-checks) * All six machines can be reached by SSH from the deployer controller. * Hostnames, IP addresses, NIC names, NTP, DNS, and firewall rules are fixed and recorded. * The four VIPs are unused and can move between their candidate nodes. * Disk mount points are fixed and survive reboot. * Enterprise, PostgreSQL, Redis, RustFS, Keepalived, HAProxy, repmgr, Fleet Agent, and rsync configurations are generated by the deployer and reviewed. * The Cluster License is activated once through the Business VIP. * Backup and restore steps are rehearsed before production traffic is introduced. ## Implementation Sequence [Section titled “Implementation Sequence”](#implementation-sequence) 1. Prepare the deployer controller and the six machines. 2. Confirm IP, VIP, NIC, disk, DNS, NTP, SSH, and firewall information. 3. Upload the Enterprise offline package and configuration material. 4. Deploy PostgreSQL primary/standby and Witness. 5. Deploy Redis independent instances and Redis VIP. 6. Deploy RustFS four-node cluster and RustFS VIP. 7. Deploy Enterprise on D/E/F, Fleet Agent, rsync filesets, and Business VIP. 8. Activate License through the Business VIP. 9. Run functional validation and failover drills. 10. Record final topology, passwords/keys, backup policy, monitoring targets, and recovery procedures. ## Acceptance Criteria [Section titled “Acceptance Criteria”](#acceptance-criteria) ### Functional Acceptance [Section titled “Functional Acceptance”](#functional-acceptance) * Users can access Enterprise through the Business VIP. * Enterprise reads and writes PostgreSQL only through PostgreSQL VIP. * Enterprise uses Redis only through Redis VIP. * Enterprise uses RustFS only through RustFS VIP. * App, Flow, Marimo, Notebook, and License files are present on the Active node after failover. * Branch Enterprise can still be installed independently and does not join this six-machine HA group. ### Failover Drill Acceptance [Section titled “Failover Drill Acceptance”](#failover-drill-acceptance) * Stop the Active business node: Business VIP moves to a healthy Standby and the business group starts only there. * Stop PostgreSQL Primary: Standby is promoted and PostgreSQL VIP continues pointing to the writable Primary. * Stop the Redis VIP owner: Redis VIP moves to the other instance, with cache/session loss accepted. * Stop the RustFS proxy owner: RustFS VIP moves to the other proxy. * Stop one RustFS data node: object access remains within the actual RustFS redundancy boundary. * Recover failed nodes: they return as Standby or healthy members and do not overwrite newer data or files. ## Risks and Items to Confirm On Site [Section titled “Risks and Items to Confirm On Site”](#risks-and-items-to-confirm-on-site) * Whether the network allows VRRP protocol `112`. * Whether the four VIP addresses are truly unused. * Whether the disk mount layout and capacity meet actual file retention and RustFS redundancy requirements. * Whether Redis only stores disposable data. * Whether PostgreSQL replication lag and backup policy meet the business RPO. * Whether operations staff understand the difference between VIP failover, file synchronization, database replication, and object storage redundancy. ## Conclusion [Section titled “Conclusion”](#conclusion) The current plan centers on six machines, four fixed VIPs, and single-active Enterprise. It decouples the business entry, database entry, cache entry, and object storage entry. It can cover common single-machine failures within the same network segment and significantly reduces the work required after failover to change connection configuration, reactivate License, and manually copy App files. At the same time, three boundaries must be accepted: Redis does not replicate data, business filesets are synchronized asynchronously, and network partition has no consensus arbitration. Therefore, this plan is suitable to define as a **Fleet Center six-machine single-active active-standby disaster recovery plan**. It should not promise zero interruption, zero data loss, or strict cross-data-center high availability.

# Standard Deployment Plan

> Reference four-VM standard deployment plan for Tier0 Enterprise.

## Objective [Section titled “Objective”](#objective) The simplified edition does not use Docker Swarm. It deploys one `enterprise` application node with a total of `4` virtual machines: * `1` enterprise node hosts `tier0-enterprise + EMQX`. * `2` database nodes form a PostgreSQL + TimescaleDB primary/standby cluster. * `1` Witness node participates in database supervision, quorum, and failover. The simplified edition reduces deployment complexity. The tradeoff is that the application layer and EMQX are single points of failure. Only the database layer keeps `DB VIP + Witness` high availability. ## Planned IP Addresses [Section titled “Planned IP Addresses”](#planned-ip-addresses) The following IP addresses are placeholders. Replace them with the real site network before implementation. | Type | Name | Planned IP | Description | | ---------------- | ------------------- | ------------- | ------------------------------------------------------------- | | Application node | `ent-enterprise-01` | `10.60.10.11` | Single enterprise node hosting `tier0-enterprise + EMQX` | | Logical entry | `DB VIP` | `10.60.20.10` | Unified database endpoint, points only to the current primary | | Database node | `ent-db-01` | `10.60.20.11` | PostgreSQL + TimescaleDB primary candidate | | Database node | `ent-db-02` | `10.60.20.12` | PostgreSQL + TimescaleDB standby candidate | | Quorum node | `ent-db-witness-01` | `10.60.20.13` | `repmgr witness` / database quorum supervisor | ## Overall Topology [Section titled “Overall Topology”](#overall-topology) ![](https://enterpriseimage.oss-cn-hangzhou.aliyuncs.com/9.png) Core paths: * Business path: clients / site systems -> `ent-enterprise-01` -> `DB VIP` -> current primary DB. * `tier0-enterprise` and `EMQX` both run on `ent-enterprise-01`. * Database access still goes through `DB VIP`; direct application access to `ent-db-01/02` is prohibited. * `Witness` participates only in database quorum, promotion, and health checks. It is not on the business path. ## Database Supervision and Failover [Section titled “Database Supervision and Failover”](#database-supervision-and-failover) | Phase | Path | Description | | -------------------- | ------------------------------------------------ | --------------------------------------------------------- | | Normal access | `ent-enterprise-01 -> DB VIP -> current primary` | Application always connects to `10.60.20.10:5432` | | Normal replication | `current primary -> current standby` | PostgreSQL WAL streaming replication | | Normal supervision | `DB-01/DB-02 -> Witness` | `repmgrd` reports status; Witness participates in quorum | | Primary failure | `standby -> Witness` | Standby requests quorum confirmation to avoid split-brain | | Standby promotion | `standby promote` | Standby is promoted after Witness confirmation | | VIP floating | `DB VIP -> new primary` | `10.60.20.10` binds to the new primary | | Application recovery | `ent-enterprise-01 -> DB VIP -> new primary` | Application recovers after connection-pool reconnect | ## Node Configuration [Section titled “Node Configuration”](#node-configuration) | Node | Role | vCPU | Memory | Storage | System disk recommendation | Data disk recommendation | Main components | | ------------------- | --------------------------------- | ---- | ------ | ------- | ---------------------------- | --------------------------------------------------- | ---------------------------------------------- | | `ent-enterprise-01` | Single enterprise node | `8` | `32G` | `100G` | Single `/` system disk, 100G | No separate data disk | tier0-enterprise, EMQX, optional Nginx/HAProxy | | `ent-db-01` | PostgreSQL/TSDB primary candidate | `4` | `8G` | `500G` | `/` 80G | `/pgdata` 360G, `/pgwal` 40G, `/backup-staging` 20G | PostgreSQL, TimescaleDB, repmgr, keepalived | | `ent-db-02` | PostgreSQL/TSDB standby candidate | `4` | `8G` | `500G` | `/` 80G | `/pgdata` 360G, `/pgwal` 40G, `/backup-staging` 20G | PostgreSQL, TimescaleDB, repmgr, keepalived | | `ent-db-witness-01` | Witness / quorum supervisor | `2` | `4G` | `40G` | `/` 40G | Not planned separately | repmgr witness, monitoring agent, audit logs | ## Configuration Summary [Section titled “Configuration Summary”](#configuration-summary) | Category | Count | Per-node spec | vCPU subtotal | Memory subtotal | Storage subtotal | | ------------------------------ | ----- | ----------------- | ------------- | --------------- | ---------------- | | enterprise application node | `1` | `8C / 32G / 100G` | `8C` | `32G` | `100G` | | PostgreSQL + TimescaleDB nodes | `2` | `4C / 8G / 500G` | `8C` | `16G` | `1000G` | | Witness node | `1` | `2C / 4G / 40G` | `2C` | `4G` | `40G` | | Total | `4` | - | `18C` | `52G` | `1140G` | Converted total: * Total VMs: `4` * Total vCPU: `18` * Total memory: `52G` * Total storage: `1.14T` ## Connectivity Matrix [Section titled “Connectivity Matrix”](#connectivity-matrix) ### Operations Access [Section titled “Operations Access”](#operations-access) | Target | Method | Port | Description | | ------------------- | --------------------------------- | ---------- | ---------------------------------------------------------------------------- | | All VMs | `ssh -i ~/.ssh/chen-rsa ops@<IP>` | `22/TCP` | Use after unified key-based SSH is implemented; use site account before that | | `ent-enterprise-01` | `https://10.60.10.11` | `443/TCP` | Business entry | | `DB VIP` | `psql -h 10.60.20.10 -p 5432` | `5432/TCP` | Allow only enterprise node and operations whitelist | ### Application and EMQX Ports [Section titled “Application and EMQX Ports”](#application-and-emqx-ports) | Source | Target | Port | Purpose | | ---------------------- | ------------------- | ----------- | ----------------------------------------- | | Users / site systems | `ent-enterprise-01` | `80/TCP` | HTTP entry or HTTPS redirect, optional | | Users / site systems | `ent-enterprise-01` | `443/TCP` | HTTPS business entry | | Devices / site systems | `ent-enterprise-01` | `1883/TCP` | EMQX MQTT plaintext, optional | | Devices / site systems | `ent-enterprise-01` | `8883/TCP` | EMQX MQTT TLS, recommended | | Devices / site systems | `ent-enterprise-01` | `8083/TCP` | MQTT WebSocket, optional | | Devices / site systems | `ent-enterprise-01` | `8084/TCP` | MQTT WebSocket TLS, optional | | Operations network | `ent-enterprise-01` | `18083/TCP` | EMQX Dashboard, operations whitelist only | | `ent-enterprise-01` | `DB VIP` | `5432/TCP` | PostgreSQL business access | ### Database and Supervision Communication [Section titled “Database and Supervision Communication”](#database-and-supervision-communication) | Source | Target | Port | Purpose | | -------------- | ------------------- | ---------- | --------------------------------------- | | `ent-db-01` | `ent-db-02` | `5432/TCP` | PostgreSQL WAL streaming replication | | `ent-db-02` | `ent-db-01` | `5432/TCP` | Reverse replication after recovery | | `ent-db-01/02` | `ent-db-witness-01` | `5432/TCP` | repmgr metadata / witness communication | | `ent-db-01` | `ent-db-02` | `VRRP` | DB VIP floating; allow protocol `112` | ## Deployment Recommendations [Section titled “Deployment Recommendations”](#deployment-recommendations) ### enterprise Application Node [Section titled “enterprise Application Node”](#enterprise-application-node) * Do not deploy Docker Swarm. * Deploy `tier0-enterprise` and `EMQX` on the same `ent-enterprise-01` node. * `ent-enterprise-01` is an application-layer single point of failure. If it fails, recover manually or rebuild from backup. * The node has only one `100G` system disk and is not suitable for long-term business data. * `tier0-enterprise` business data must be stored in PostgreSQL/TimescaleDB. * Uploads, imports/exports, and reports may be stored locally and backed up. If later upgraded to the standard edition, migrate these files to NFS/NAS or object storage. ### Single-node EMQX [Section titled “Single-node EMQX”](#single-node-emqx) * EMQX is a single-node stateful service in the simplified edition. * Pin its data directory on `ent-enterprise-01`, for example `/data/emqx`. * Back up configuration, certificates, plugins, and rule-engine configuration. * No message-layer HA is provided. Use the standard edition with a three-node EMQX cluster if this is unacceptable. ### PostgreSQL + TimescaleDB [Section titled “PostgreSQL + TimescaleDB”](#postgresql--timescaledb) * PostgreSQL: `16` or the site-certified version. * TimescaleDB: stable version compatible with the selected PostgreSQL major version. * Replication: physical streaming replication. * Failover: `repmgrd + witness`. * Database entry: `DB VIP`. * Connection pool: PgBouncer can run on `ent-enterprise-01` to reduce database connection pressure. ### Witness [Section titled “Witness”](#witness) * Witness stores only `repmgr witness` metadata and supervision logic. * Witness does not host business SQL. * Witness failure does not stop the current primary, but reduces trusted automatic failover capability. ## Port and Firewall Baseline [Section titled “Port and Firewall Baseline”](#port-and-firewall-baseline) | Port / protocol | Allowed scope | Purpose | | --------------- | --------------------------------------------- | ----------------------------------------- | | `22/TCP` | Operations network to 4 VMs | SSH | | `80/TCP` | User network to `ent-enterprise-01` | HTTP redirect or internal plaintext entry | | `443/TCP` | User network to `ent-enterprise-01` | HTTPS business entry | | `1883/TCP` | Devices / site systems to `ent-enterprise-01` | EMQX MQTT plaintext, optional | | `8883/TCP` | Devices / site systems to `ent-enterprise-01` | EMQX MQTT TLS, recommended | | `8083/TCP` | Devices / site systems to `ent-enterprise-01` | EMQX WebSocket, optional | | `8084/TCP` | Devices / site systems to `ent-enterprise-01` | EMQX WebSocket TLS, optional | | `18083/TCP` | Operations network to `ent-enterprise-01` | EMQX Dashboard, whitelist only | | `5432/TCP` | enterprise to `DB VIP`; DB/Witness internal | PostgreSQL / repmgr | | `112/VRRP` | Between DB nodes | DB VIP floating | | `9100/TCP` | Monitoring system to 4 VMs | node exporter, optional | ## HA Capabilities and Failure Boundaries [Section titled “HA Capabilities and Failure Boundaries”](#ha-capabilities-and-failure-boundaries) | Failure scenario | Expected behavior | Business impact | | ------------------------- | ---------------------------------------------------------------------------------- | --------------------------------------------------------------------- | | `ent-enterprise-01` fails | Application and EMQX both stop; manual recovery or rebuild from backup is required | Full application-layer outage | | Current DB primary fails | Standby is promoted after Witness quorum; DB VIP floats | Write interruption for seconds to minutes; application must reconnect | | DB standby fails | Primary continues serving, replication alert is raised | No direct business outage, but DB hot standby is lost | | Witness fails | Current primary continues serving, but automatic failover confidence is reduced | No direct business outage, restore Witness first | | DB VIP floating fails | Database may have a new primary, but unified entry is unreachable | Business outage; manual VIP binding required | ## Pre-Go-Live Checklist [Section titled “Pre-Go-Live Checklist”](#pre-go-live-checklist) | Check item | Pass criteria | | ----------------------------- | ----------------------------------------------------------------- | | 4 VM hostnames are fixed | `hostnamectl` matches the plan | | Unified SSH works | Operations terminal can access all 4 VMs with key-based SSH | | Time synchronization | Time drift across 4 VMs is below `1s` | | Firewall is open | Ports from section 9 are allowed by scope | | enterprise service is healthy | `tier0-enterprise` health check passes | | EMQX state is verified | EMQX node, listening ports, and persistent directory are healthy | | enterprise disk usage is safe | `/` usage is below `70%`; log and cache cleanup is configured | | DB primary/standby is healthy | `repmgr cluster show` shows primary, standby, and witness healthy | | DB VIP works | enterprise node can connect to `10.60.20.10:5432` | | Primary DB failover drill | Standby can be promoted and DB VIP can float | | Backup restore drill | Backup can be restored to a test database | ## Implementation Order [Section titled “Implementation Order”](#implementation-order) 1. Create 4 VMs and fix hostnames, IPs, time synchronization, and SSH. 2. Initialize OS baseline, firewall, mounts, and kernel parameters. 3. Deploy `tier0-enterprise` on `ent-enterprise-01`. 4. Deploy single-node EMQX on `ent-enterprise-01`. 5. Deploy PostgreSQL + TimescaleDB primary/standby. 6. Deploy repmgr witness, repmgrd, and DB VIP floating scripts. 7. Verify connectivity, EMQX recovery, database failover, backup restore, and routine checks. 8. Finalize runbooks, credential archives, and routine check tasks. ## Risks and Open Items [Section titled “Risks and Open Items”](#risks-and-open-items) * The simplified edition has an application-layer single point. If `ent-enterprise-01` fails, the whole business service is unavailable. * EMQX is single-node and does not provide message-layer HA. * `ent-enterprise-01` has only a single `100G` system disk. Logs, cache, attachments, and temporary files must be controlled. * If application-layer or EMQX HA is required later, switch to the standard edition. * Database RPO/RTO is not confirmed. This primary/standby design cannot guarantee zero data loss. * DB VIP floating scripts must be verified on site, especially whether the virtualization/network platform allows VRRP.

# User Management

> Tier0 uses membership, licenses, and custom user accounts to let customers manage platform access and user permissions.

Tier0 manages user permissions and access differently across editions. * Cloud uses Workspace member invitations to manage who can access a Workspace. * Enterprise admin assigns user accounts and roles to determine correspondingly accessible functions. ## Access Control [Section titled “Access Control”](#access-control) * Cloud * Workspace owner or existing members can invite new members by phone or email. * After receiving an invitation code or link, new members register an account and join the Workspace. * Account registration requires two-step verification. Caution Workspace membership does not automatically grant access to the platform; users must register the invited phone or email to use the platform. * Enterprise * After signing in for the first time with the default account, the admin changes the password, then adds user accounts and assigns roles in **Settings**. * Admin determines who can use the platform by assigning accounts, and which functions the account can access by assigning roles: * **Admin**: Can access all functional modules. * **Operator**: Can only access the **Launchpad** where all accessible applications are listed.