Do you feel a sense of guilt when you take time to go on a date with you wife?
Does guilt grab ahold of you if you are playing in the backyard with your children and a church member drives by?
If that church member stops to say hello, does a feeling of guilt lead you to give a reason or excuse for taking time to enjoy your children?
Does guilt consume you while on vacation?
When the day is done, do you deal with guilt because you didn’t get all of your boxes checked or those visits made?
GUILT….an emotion that can create ongoing discouragement in ministry.
How do you deal with it?
1. If you have started your day in prayer seeking the Lord’s will as well as surrendering your plans to His leading, then rest in His sovereign care and direction.
The steps of a good man are ordered by the Lord, and He delights in his way. Though he fall, he shall not be utterly cast down; for the Lord upholds him with His hand (Psalm 37:23-24).
The plans of the heart belong to man, but the answer of the tongue is from the Lord. Commit your work to the Lord, and your plans will be established. The Lord has made everything for its purpose (Proverbs 16:1, 3-4a).
2. Live to please the Lord, to glorify Him, and realize that you will never be able to please everyone in your congregation. Furthermore, they cannot always please you either. Yes, you are an example to the flock and they are watching you, but if you live to honor the Lord (1 Corinthians 10:31), then you have lined up with the highest standard. So respond with humility and grace. Neither pastor nor parishioner should have a standard for the other that they themselves are not living up to. Be gracious and give others some latitude.
3. Your freedom from the fear of man will be a testimony to your people as you interact with them in love and compassion. Lead them to see that for all of us, our expectation, our hope must come from the Lord (Psalm 62:5).
The fear of man lays a snare, but whoever trusts in the Lord is safe (Proverbs 29:25).
4. Your identity is in Christ alone in Whom you are accepted. Spend some time soaking your soul in Romans 6-8 and Philippians 1.
To the praise of the glory of His grace, by which He made us accepted in the Beloved (Ephesians 1:6).
I have been crucified with Christ; it is no longer I who live, but Christ lives in me; and the life which I now live in the flesh I live by faith in the Son of God, who loved me and gave Himself for me (Galatians 2:20).
Ministry friend, remember that we live in a sin-cursed world that will never work right. So, rest in the promises of God, rest in grace and knowledge of the Lord Jesus Christ, and rest in Matthew 11:28-30, Come to Me, all you who labor and are heavy laden, and I will give you rest. Take My yoke upon you and learn from Me, for I am [f]gentle and lowly in heart, and you will find rest for your souls. For My yoke is easy and My burden is light.
From A Pastor's Heart 
All of us have been there, brother. In my second church, I had a lady who watched my house day and night and grilled me about any visitors I might have had or how long my lights were on at night. I have often thought over these nearly 50 years of ministry that if people spent as much time looking unto Christ as they did examining my life I would have had really godly congregations. But I realized this very early working with your dad, if the pastor feeds the flock instead of beating them, ultimately they will be content and it does not matter what men and women think of your daily activities