Apps are externally (outside of Shopify) hosted applications that interact with stores. A private app can poll products, customers, orders, etc. It can also add or remove products and customers, change metafields, and adjust orders. Private apps can set up 'webhooks' so that stores ping them when certain events occurs, such as new customers, new orders, a change in product inventory, etc. Basically, you are right in that an app is basically remote access to a Shopify store, but because it does things programmatically it can automate large tasks, synchronize online inventory with a separate database, or do other things that it would take people a long time to do in the admin.
In addition, apps can edit store themes and add scripts to a store's frontend so there's a way to add store functionality via an app that way. Basically, most of the things you can do by hand in the admin can be done by apps.
Perhaps if you tell us what it is you are looking to do via an app, we can help.