Problem #1: My validator has
Your validator has become jailed. Validators get jailed, i.e. get removed from the active validator set, if they do not
500 of the last
10000 blocks, or if they double sign.
If you got jailed for downtime, you can get your voting power back to your validator. First, if
desmos is not running,
start it up again:
Wait for your full node to catch up to the latest block. Then, you can unjail your validator.
Lastly, check your validator again to see if your voting power is back.
You may notice that your voting power is less than it used to be. That's because you got slashed for downtime!
Problem #2: My
desmos crashes because of
too many open files
The default number of files Linux can open (per-process) is
desmos is known to open more than
This causes the process to crash. A quick fix is to run
ulimit -n 4096 (increase the number of open files allowed) and
then restart the process with
desmos start. If you are using
systemd or another process manager to launch
this may require some configuration at that level. A sample
systemd file to fix this issue is below:
Description=Desmos Full Node
User=ubuntu # This is the user that is running the software in the background. Change it to your username if needed.
WorkingDirectory=/home/ubuntu # This is the home directory of the user that running the software in the background. Change it to your username if needed.
ExecStart=/home/ubuntu/go/bin/desmos start # The path should point to the correct location of the software you have installed.
LimitNOFILE=4096 # To compensate the "Too many open files" issue.
Problem #3: My validator is inactive/unbonding
When creating a validator you have the minimum self delegation amount using the
--min-self-delegation flag. What this
means is that if your validator has less than that specific value of tokens self delegated, it will automatically enter
the unbonding state and then be marked as inactive.
To solve this, what you can do is getting more tokens delegated to it by following these steps:
Get your address:
desmos keys show <your_key> --address
Require more tokens using the Discord bot inside the
#ask-tokenschannel by sending
!send <address>. (TESTNET ONLY)
Make sure the tokens have been sent properly:
desmos query account $(desmos keys show <your_key> --address) --chain-id <chain_id>
Delegate the tokens to your validator:
desmos tx staking delegate \
$(desmos keys show <your_key> --bech=val --address) \
--chain-id <chain_id> \
--from <your_key> --yes
# desmos tx staking delegate \
# $(desmos keys show validator --bech=val --address) \
# 10000000udaric \
# --chain-id morpheus-apollo-2 \
# --from validator --yes
Problem #4: My validator is jailed
If your validator is jailed it probably means that it has been inactive for a long period of time missing a consistent number of blocks. We suggest you checking the Desmos daemon status to make sure it hasn't been interrupted by some error.
If the service is running properly, it probably means that your node did not have internet access for a prolonged period of time. In both cases, if there are no other errors to fix, you can unjail your validator by executing the following command:
desmos tx slashing unjail --chain-id <chain_id> --from <your_key>
# desmos tx slashing unjail --chain-id morpheus-apollo-2 --from validator
This will perform an unjail transaction that will set your validator as active again from the next block.
If the problem still persists, please make sure you have enough tokens delegated to your validator.
If your service is running properly, you can also try and reset your
desmos configuration by running the following
After doing so, remember to restart your validator service to apply the changes:
systemctl restart desmosd
Problem #5: The persistent peers do not work properly
Sometimes, it might happen that your node cannot connect to the persistent peers we have provided inside the testnet repository. This happens because all nodes have a limit of inbound connections that they can accept. Once that limit is exceed, the nodes will not accept any more connections.
In order to solve this problem, there are two alternative way:
- use a seed node instead of a persistent peer, OR
- use different persistent peers.
Using a seed node
Seed nodes are a particular type of nodes that provide every validator with a set of peers to connect with, based on the current network status. What will happen when you use seed nodes is the following:
- Your node will connect to a seed node.
- The seed node will provide your node with a list of peers.
- Your node will disconnect from the seed node and connect to the peers.
- Your node will start syncing with the chain.
In order to use this particular type of nodes, all you have to do is:
Find the line starting with
seeds = ""
Replace that line with the following:
Empty your persistent peers list by replacing the
pesistent_peers = "..."line with
persistent_peers = ""
Save the file and exit the editor.
Restart your node.
Changing your persistent peers
Instead of using a seed node, you can also keep relying on persistent peers. In this case, you will need to find new ones to connect your node to. To do this, you can query the current peers of any chain node using the following RPC endpoint:
For example, you can use the public RPC endpoint here.
From that page, you can see all the peers connected to that node. Their info is present inside the
peers field, which contains a list of objects made as follows:
In order to get new peers addresses, all you have to do is to combine the
listen_addr field values as follows:
In the above case, that peer's address would be:
You can do this with as many peers as you want. Once you have a list of peers, you can use those inside
persistent_peers field of your
Problem #6: I tried unjailing my validator, but it keeps getting jailed after some time
If you have tried to unjail, but you've seen that your node is jailed again shortly after, it most probably means that your validator has been tombstoned.
A validator is put in a tombstone status only when it double signs. Since this is way more egregious than a liveliness fault, once your validator double signs it will no longer be able to re-join the active set with the same validator key.
In order to avoid this, you need to always make sure that each of your nodes do not validate with the same private key.
Also, once your validator is tombstoned all you can do is create a new one, and earn again all the delegations that you had before.
Problem #7: My validator run out of space
This situation occurs if you haven’t allocated enough disk space to your node when you’ve created it. The space you need is directly related to the pruning strategy your using, you can read more about it here.
Assuming you’re using a VPS, to solve this we can rely on different strategies, each of one of them with upside/downside:
A) Add more diskspace and extend your filesystem to use it: 1) Log into the provider console and buy more diskspace; 2) Follow this guide to learn how to extend the filesystem on linux; 3) Restart your validator node.
Pros: Faster solution, ideal for mainnet validators.
Cons: Raise renting costs of VPS.
B) Switch pruning strategy, reset your node, state-sync it:
1) Stop your node daemon service (usually sudo
systemctl stop desmosd);
2) Navigate to
.desmos/config/ and open app.toml;
3) Switch from pruning nothing to default/everything or from default to everything*;
4) Backup the
addrbook.json file (this will help the node connect faster to peers after the restart);
5) Navigate to .desmos/data/ and backup the
priv_validator_state.json file (this will keep the voting state avoiding double sign);
6) Unsafe reset your node with
7) Place the backup of
priv_validator_state.json back into
.desmos/data/ folders respectively;
8) Resync the node with state-sync (if possible).
Pros: Cheaper solution, help to understand the meaning of different pruning strategies.
Cons: Not feasible for mainnet validators, longer times to be back online validating.
*About pruning everything:
Currently, pruning everything looks to be unsafe and unstable so we suggest to NOT use this strategy in production.
However, our team is currently testing it inside our morpheus-apollo-2 testnet and will give some results in the upcoming weeks/months.
Problem #8: Wrong Block Header AppHash
If you run into this problem while you're syncing your node, probably you are using a wrong Desmos binary version. If so, please ask inside our discord server what's the correct version/versions to use in order to sync the node correctly.
Previous delegators will still be able to unbond from a tombstoned validator.
More information about the slashing penalties and tombstoning can be found here.