Welcome to the forum. The best way to accomplish it on the cheap is to look at lighter tires and tubes since most OEM are heavy. This is also after you have worn them out. Same with the chain, bar tape and cable and housing. Also the stock saddle and bars may not agree with you and if you do have to change them look for lighter options. If you replace as things wear out it was have less impact since you have to replace components anyway.
Last words, replace rotating parts first and the largest on down. Tires, tubes and lighter wheels will have the greatest impact on your bike.