Prepare a straight-sided 9 by 13-inch pan by lining it with plastic wrap or foil. Set aside.
Put the peanut butter in a small microwavable bowl and microwave for about 30 seconds. Stir and set aside. It should be a pourable consistency.
Remove the ice cream from the freezer and allow it to soften slightly, about 5 minutes. Dump it into a large bowl and stir until it's the consistency of thick buttercream frosting.
Pour the peanut butter into the ice cream and begin folding it throughout, not overmixing. Then add the jelly and do the same.
Pour the ice cream into the prepared pan and smooth the top with an angled spatula. Cover it with another piece of plastic wrap and place the pan in the freezer for at least 4 hours or until the ice cream is solid.
MAKE THE COOKIE DOUGH
Whisk the gluten free flour blend, baking powder, and salt in a medium bowl until well combined. Set aside.
Cream butter with sugars until light and fluffy, about 4 minutes. Add eggs and vanilla and continue to beat until fluffy, about 2 minutes. Slowly add the flour mixture on low speed and pulse just until combined. Use a spatula to incorporate any dry ingredients left in the bottom of the bowl.
Divide the dough in half and wrap each in plastic wrap. Refrigerate for 2 hours.
BAKE THE COOKIES
Preheat the oven to 350° F. Remove the dough from the refrigerator and allow it to sit for about 5 minutes to soften slightly. Divide the dough into thirds and color one third with red food coloring (no taste if you can find it) and another with blue. Leave the third with no color. Set them side by side, red, white, and blue, and roll them out together.
Cut out with star cookie cutter and place cookies on a parchment-lined baking sheet, about ½-inch apart (they won't spread). Chill the cookies for 10 minutes. Then bake at 350° F for about 12 minutes. They should be just set, but not golden around the edges. Let cookies cool on baking sheets on wire racks.
ASSEMBLE ICE CREAM SANDWICHES
Remove the ice cream from the freezer and, using the same cookie cutter, cut out the star shapes and place them on overturned cookies. Top with another cookie and press lightly to adhere.
Wrap sandwiches in plastic wrap and freeze overnight, until ice cream is hard and the cookies are slightly softened. The longer the cookies sit in the freezer, the more the cookies will soften. Ice cream sandwiches will keep in the freezer, well wrapped, for up to three months.