A person who refrains from bestowing kindness to a poor person should be very fearful that perhaps that poor person will cry out to Hashem about him, as it is written, If he then complains to God about you, you will have a sin (Devarim 15:9). This can cause the wealthy person’s entire financial situation to be reversed. Our Sages thus said: When a poor person approaches a man of wealth and begs “support me” — if the man gives him, fine and well; but if he does not give him, about such a person it is written, The rich man and the pauper meet; Hashem is the Maker of them all (Mishlei 22:2). The One Who made this person wealthy will make him poor, and the One Who made that person poor will make him wealthy.
The Sages similarly expounded the verse, Because you did not serve Hashem, your God, amid gladness and goodness of heart, when everything was abundant, so you will serve your enemies whom Hashem will send against you, in hunger and in thirst, in nakedness and without anything (Devarim 28:47-48); that is, when a person of means does not use his wealth to extend kindness towards his poor brethren, which would have been a fulfillment of a Torah-level Mitzvah and would not have caused him any financial loss (because whatever he would have given would have come back to him many times over), he suffers instead that all his possessions are taken away from him at the hands of gentiles, from whom he will never be able to retrieve them.
And even if does give some small amount to charity and performs some small acts of kindness, if what he gave or did does not match what he was capable of giving or doing, he is similarly at risk of losing all his wealth as a result. This risk even extends to other people’s money which were deposited to his care; that money as well might be lost.